RFC-38: Events filtering and batching for Forge

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building shared understanding of a topic. It expresses a technical solution, but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

RFC dates:

  • Publish: March 4, 2024
  • Discuss: March 18, 2024
  • Resolve: March 22, 2024

Summary of Project

We’re in the early stages of planning an event filtering and batching feature for Forge apps and we’d like to ask you about your needs and expectations in this area before we start the development.

Event filtering is a functionality which will give apps the ability to create criteria for, what kind of events they should receive. For example: instead of listening to all issue:updated events from the instance, you will be able to listen to events only from specific projects.

Events batching is a functionality which will group product events into batches and invoke a single function for a group of events. For example: instead of 10 invocations, you will receive one invocation with data from 10 events.

Problem

Currently, Forge app developers who listen to product events are running into problems when there is an influx of events on the customer’s site. There is no way of listening to only the events that are needed for their apps, causing costs both on their side and our side. Moreover, it reduces the amount of business logic on the app side and decreases the amount of unnecessary traffic on the product side.

We also believe that introducing batched events it will enable partners to reduce the processing time and increase Forge performance due to the usage of bulk APIs.

Proposed solution

At the moment we do not have a chosen solution although we are considering the following approaches:

  • Introduce a completely new way of filtering for Forge apps that won’t be Jira specific.
  • Introduce as a first step events filtering for Jira Forge apps only similar to what we have currently in Connect - JQL based or use another filtering method like Jira Expressions.
  • Introduce batching by event type with a max events and batching window limits.

Asks

We are still early in the design/proof-of-concept phase and are keen for your feedback. While we would appreciate any feedback, we’re especially interested in learning more about:

Events filtering:

  • What use cases do you have for filtering? Which products and fields do you want to filter events by?
  • Is the current set of fields that’s been sent in the payload enough to filter events?
  • What resources do you anticipate filtering? Are there well-established expression languages for filtering events about those resources?
  • Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

Events batching

  • What use cases do you have for batching?
    • What’s the longest batch window that’d work for you? Are you willing to trade throughput for latency? Why?
    • What configuration options do you need for batching?
    • Do you have any requirements around minimum/maximum batch size?

Of course, we’d also appreciate it if you would share any other thoughts about event filtering and batching and Connect/Forge webhooks in general.

Thank you.

15 Likes

Thanks for sharing @gkaszkur,

This sounds like a very nice feature!

What use cases do you have for filtering? Which products and fields do you want to filter events by?

We would like to filter Confluence events. Specifically, for us it would be nice to be able to only receive events for pages that are:

  • Part of specific space.
  • Descendants of a specific page.
  • Have a specific content property.
  • Descendants of a page with a specific content property.

There might be more cases.

What resources do you anticipate filtering? Are there well-established expression languages for filtering events about those resources?

If by that you mean JQL / CQL, that would certainly be a good start and would enable a good amount of use cases. There might be more sophisticated ones where these languages are not enough, though.

Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

For the use cases of our products, we would almost certainly have the registration to be dynamic. For example, with one of our apps you can publish a specific Confluence space as a website. If you do that, we would like to listen to all events for that space to be able to update the website accordingly. Since this space is only configured by the user at runtime, we’d also need to perform the event listener registration at runtime.

What use cases do you have for batching?

This could simplify certain use cases such as making it less likely to receive certain events out of order (which can be problematic). But in general I think the main benefit for this would just be that we receive a lower amount of network requests in total.

What’s the longest batch window that’d work for you? Are you willing to trade throughput for latency? Why?

With our website use case, we’d want changes to go live as fast as possible. So anything over 10 seconds would probably already be rather uncomfortable for us. In fact, we’d probably rather have a lower latency for our use case.

What configuration options do you need for batching? Do you have any requirements around minimum/maximum batch size?

Perhaps, if we could configure the latency ourselves that would be great. We would not have any requirements for min/max batch sitze at the moment, and probably no need for other configuration options at the moment - but it would probably still be good to be able to configure batch size.

Cheers,
Sven

5 Likes

Hi @gkaszkur !

Thanks for this RFC - we really look forward to use this to improve our app!

Events filtering

What use cases do you have for filtering? Which products and fields do you want to filter events by?

We would like to filter for Jira events on changes to issues for a set of fields, including custom fields and also changes to issue links.

Is the current set of fields that’s been sent in the payload enough to filter events?

Assuming that this is about Jira events we would like to specify a set of issue fields (including custom fields) and also issue links.

What resources do you anticipate filtering? Are there well-established expression languages for filtering events about those resources?

Our app uses e.g. Jira issue filters to determine the content, thus JQL would be the first choice, but other languages for expressing the criterias could do too.

Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

As our app visualizes issues based on a query, dynamic filtering is a requirement. The filtering specifics (issues, fields, issue links) are known and changed at runtime and cannot be hardcoded in a manifest.

Events batching

What use cases do you have for batching?
What’s the longest batch window that’d work for you? Are you willing to trade throughput for latency? Why?
What configuration options do you need for batching?
Do you have any requirements around minimum/maximum batch size?

Receiving event batches every 10 seconds would probably do just fine.

Thanks,
Fredrik

4 Likes

I’ll point out GitHub - aws/event-ruler: Event Ruler is a Java library that allows matching many thousands of Events per second to any number of expressive and sophisticated rules. - this is the language/engine that AWS uses within its widely used and loved EventBridge service:

Event Ruler […] is a Java library that allows matching Rules to Events. An event is a list of fields, which may be given as name/value pairs or as a JSON object. A rule associates event field names with lists of possible values. There are two reasons to use Ruler:

  1. It’s fast; the time it takes to match Events doesn’t depend on the number of Rules.
  2. Customers like the JSON “query language” for expressing rules.

The announcement post Open Sourcing Event Ruler provides a few more details on features, performance, and internals:

Today, Amazon is excited to open source a key technology that powers EventBridge as Event Ruler under the Apache 2.0 license. It is a Java library that allows you to build applications that can match any number of rules against events at several hundred thousand events per second. Both events and rules are JSON objects, but rules additionally can be expressed through an inbuilt query language that helps describe custom matching patterns.

4 Likes

For our event handlers, the first thing we always do is to reload the issue again, because the one supplied doesn’t contain the details we need. So we want to be able to specify the parameters for the issue query (specifying fields, rendered…)

4 Likes

Hi @gkaszkur! We would prefer a solution that also works with Confluence and not just with Jira. Our apps store a lot of information in space and content properties, so a filter definition that can use these properties would be really useful. For example, we would like to receive the avi:confluence:updated:page event only if a property of the space that contains the page has a certain value.

3 Likes

Hello there,

First of all, thanks for sharing this RFC with all of us!

Events filtering:

Actually one of our Forge apps is filtering issues by project type.

We are looking at the moment for the summary, the description and the comments so it is not enough but we understand that dealing with comments may require getting them using the API.

In our current situation it may be enough with the static approach but I’m afraid we may need more in the near future, so maybe dynamic custom fields can be the example here.

Event batching:

We are working with comments in one of our apps and analyzing them. For that, batching this event will be great so we can analyze them as together do it on their own.

A batch of some minutes will be needed for this particular use case. Having the possibility to configure the batching time will be nice.

Let me know if you will need more concrete feedback of any of these topics.

Have a nice day!

4 Likes

While I agree that a fast rule engine should be used, it also feels like “yet another way to filter” - there are already JQL and Jira expression, that are used at different parts of the system. So I wonder, for the sake of consistency, if we could make the Jira expression work that only operates on the issue?

Bye
Eckhard

We have one filtering that uses some more elaborate filtering, we filter on page properties and go to the REST endpoint, for example.

No, but I would not expect that in our case.

Dynamic one, as we have a toggle if we want to filter at all - and with this, we could easily tell “no filtering” at runtime and then evade all the extra work needed right now.

Best
Eckahrd

1 Like

Hello everyone,

We appreciate your feedback and would like to update you on some developments based on your suggestions. We understand that you are interested in dynamic filters and a focus on cross-product solutions.

Your suggested use cases for filtering, including filtering by custom fields, comments, issue links, project types or confluence content properties, amongst others, have been instrumental in shaping our approach. While there wasn’t a clear preference for a specific filter language or semantic, we’ve acknowledged the significance of flexibility in this context.

So, what’s next? We’re exploring at the moment different solutions based on your requirements. We’re primarily moving towards a cross-product solution for filtering. We’re confident in this direction, but we’re also realistic - if technical obstacles arise that we can’t get around, we’ll revisit our course of action. Our main goal is to deliver a dynamic filtering option that can be set by apps.

As we get closer to proposing a final solution, we’ll continue to keep you updated. Your feedback and participation are invaluable to us in creating a product that truly meets your needs. Therefore, we encourage you to stay alert for future communications as we continue this journey together.

Thank you for being an important part of our community. More updates will follow soon in this thread.

All the best,
Grażyna

5 Likes