RFC-38: Events filtering and batching for Forge

[gkaszkur]

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building shared understanding of a topic. It expresses a technical solution, but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

RFC dates:

• Publish: March 4, 2024
• Discuss: March 18, 2024
• Resolve: March 22, 2024

Summary of Project

We’re in the early stages of planning an event filtering and batching feature for Forge apps and we’d like to ask you about your needs and expectations in this area before we start the development.

Event filtering is a functionality which will give apps the ability to create criteria for, what kind of events they should receive. For example: instead of listening to all issue:updated events from the instance, you will be able to listen to events only from specific projects.

Events batching is a functionality which will group product events into batches and invoke a single function for a group of events. For example: instead of 10 invocations, you will receive one invocation with data from 10 events.

Problem

Currently, Forge app developers who listen to product events are running into problems when there is an influx of events on the customer’s site. There is no way of listening to only the events that are needed for their apps, causing costs both on their side and our side. Moreover, it reduces the amount of business logic on the app side and decreases the amount of unnecessary traffic on the product side.

We also believe that introducing batched events it will enable partners to reduce the processing time and increase Forge performance due to the usage of bulk APIs.

Proposed solution

At the moment we do not have a chosen solution although we are considering the following approaches:

• Introduce a completely new way of filtering for Forge apps that won’t be Jira specific.
• Introduce as a first step events filtering for Jira Forge apps only similar to what we have currently in Connect - JQL based or use another filtering method like Jira Expressions.
• Introduce batching by event type with a max events and batching window limits.

Asks

We are still early in the design/proof-of-concept phase and are keen for your feedback. While we would appreciate any feedback, we’re especially interested in learning more about:

Events filtering:

• What use cases do you have for filtering? Which products and fields do you want to filter events by?
• Is the current set of fields that’s been sent in the payload enough to filter events?
• What resources do you anticipate filtering? Are there well-established expression languages for filtering events about those resources?
• Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

Events batching

• What use cases do you have for batching?
  • What’s the longest batch window that’d work for you? Are you willing to trade throughput for latency? Why?
  • What configuration options do you need for batching?
  • Do you have any requirements around minimum/maximum batch size?

Of course, we’d also appreciate it if you would share any other thoughts about event filtering and batching and Connect/Forge webhooks in general.

Thank you.

Update (October 2, 2024)

We have recently added more information to the comment below regarding the events filtering solution. If you have any feedback for our team, please feel free to share. The RFC will remain open for the next two weeks to gather your input (October 2 - 16, 2024).

[SvenSchatter]

Thanks for sharing @gkaszkur,

This sounds like a very nice feature!

What use cases do you have for filtering? Which products and fields do you want to filter events by?

We would like to filter Confluence events. Specifically, for us it would be nice to be able to only receive events for pages that are:

• Part of specific space.
• Descendants of a specific page.
• Have a specific content property.
• Descendants of a page with a specific content property.

There might be more cases.

What resources do you anticipate filtering? Are there well-established expression languages for filtering events about those resources?

If by that you mean JQL / CQL, that would certainly be a good start and would enable a good amount of use cases. There might be more sophisticated ones where these languages are not enough, though.

Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

For the use cases of our products, we would almost certainly have the registration to be dynamic. For example, with one of our apps you can publish a specific Confluence space as a website. If you do that, we would like to listen to all events for that space to be able to update the website accordingly. Since this space is only configured by the user at runtime, we’d also need to perform the event listener registration at runtime.

What use cases do you have for batching?

This could simplify certain use cases such as making it less likely to receive certain events out of order (which can be problematic). But in general I think the main benefit for this would just be that we receive a lower amount of network requests in total.

What’s the longest batch window that’d work for you? Are you willing to trade throughput for latency? Why?

With our website use case, we’d want changes to go live as fast as possible. So anything over 10 seconds would probably already be rather uncomfortable for us. In fact, we’d probably rather have a lower latency for our use case.

What configuration options do you need for batching? Do you have any requirements around minimum/maximum batch size?

Perhaps, if we could configure the latency ourselves that would be great. We would not have any requirements for min/max batch sitze at the moment, and probably no need for other configuration options at the moment - but it would probably still be good to be able to configure batch size.

Cheers,
Sven

[freatt]

Hi @gkaszkur !

Thanks for this RFC - we really look forward to use this to improve our app!

Events filtering

What use cases do you have for filtering? Which products and fields do you want to filter events by?

We would like to filter for Jira events on changes to issues for a set of fields, including custom fields and also changes to issue links.

Is the current set of fields that’s been sent in the payload enough to filter events?

Assuming that this is about Jira events we would like to specify a set of issue fields (including custom fields) and also issue links.

What resources do you anticipate filtering? Are there well-established expression languages for filtering events about those resources?

Our app uses e.g. Jira issue filters to determine the content, thus JQL would be the first choice, but other languages for expressing the criterias could do too.

Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

As our app visualizes issues based on a query, dynamic filtering is a requirement. The filtering specifics (issues, fields, issue links) are known and changed at runtime and cannot be hardcoded in a manifest.

Events batching

What use cases do you have for batching?
What’s the longest batch window that’d work for you? Are you willing to trade throughput for latency? Why?
What configuration options do you need for batching?
Do you have any requirements around minimum/maximum batch size?

Receiving event batches every 10 seconds would probably do just fine.

Thanks,
Fredrik

[sopel]

@gkaszkur

Are there well-established expression languages for filtering events about those resources?

I’ll point out GitHub - aws/event-ruler: Event Ruler is a Java library that allows matching many thousands of Events per second to any number of expressive and sophisticated rules. - this is the language/engine that AWS uses within its widely used and loved EventBridge service:

Event Ruler […] is a Java library that allows matching Rules to Events. An event is a list of fields, which may be given as name/value pairs or as a JSON object. A rule associates event field names with lists of possible values. There are two reasons to use Ruler:

1. It’s fast; the time it takes to match Events doesn’t depend on the number of Rules.
2. Customers like the JSON “query language” for expressing rules.

The announcement post Open Sourcing Event Ruler provides a few more details on features, performance, and internals:

Today, Amazon is excited to open source a key technology that powers EventBridge as Event Ruler under the Apache 2.0 license. It is a Java library that allows you to build applications that can match any number of rules against events at several hundred thousand events per second. Both events and rules are JSON objects, but rules additionally can be expressed through an inbuilt query language that helps describe custom matching patterns.

[paul]

For our event handlers, the first thing we always do is to reload the issue again, because the one supplied doesn’t contain the details we need. So we want to be able to specify the parameters for the issue query (specifying fields, rendered…)

[klaussner]

Hi @gkaszkur! We would prefer a solution that also works with Confluence and not just with Jira. Our apps store a lot of information in space and content properties, so a filter definition that can use these properties would be really useful. For example, we would like to receive the avi:confluence:updated:page event only if a property of the space that contains the page has a certain value.

[alvaro.aranda]

Hello there,

First of all, thanks for sharing this RFC with all of us!

Events filtering:

@gkaszkur

What use cases do you have for filtering? Which products and fields do you want to filter events by?

Actually one of our Forge apps is filtering issues by project type.

@gkaszkur

Is the current set of fields that’s been sent in the payload enough to filter events?

We are looking at the moment for the summary, the description and the comments so it is not enough but we understand that dealing with comments may require getting them using the API.

@gkaszkur

Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

In our current situation it may be enough with the static approach but I’m afraid we may need more in the near future, so maybe dynamic custom fields can be the example here.

Event batching:

@gkaszkur

What use cases do you have for batching?

We are working with comments in one of our apps and analyzing them. For that, batching this event will be great so we can analyze them as together do it on their own.

A batch of some minutes will be needed for this particular use case. Having the possibility to configure the batching time will be nice.

Let me know if you will need more concrete feedback of any of these topics.

Have a nice day!

[EckhardMaass]

@sopel

I’ll point out GitHub - aws/event-ruler: Event Ruler is a Java library that allows matching many thousands of Events per second to any number of expressive and sophisticated rules. - this is the language/engine that AWS uses within its widely used and loved EventBridge service:

While I agree that a fast rule engine should be used, it also feels like “yet another way to filter” - there are already JQL and Jira expression, that are used at different parts of the system. So I wonder, for the sake of consistency, if we could make the Jira expression work that only operates on the issue?

Bye
Eckhard

[EckhardMaass]

@gkaszkur

• What use cases do you have for filtering? Which products and fields do you want to filter events by?

We have one filtering that uses some more elaborate filtering, we filter on page properties and go to the REST endpoint, for example.

• Is the current set of fields that’s been sent in the payload enough to filter events?

No, but I would not expect that in our case.

• Would you prefer static (hard-coded in the manifest) or dynamic (calculated at runtime) filtering? Which one would you use first and why?

Dynamic one, as we have a toggle if we want to filter at all - and with this, we could easily tell “no filtering” at runtime and then evade all the extra work needed right now.

Best
Eckahrd

[gkaszkur]

Hello everyone,

We appreciate your feedback and would like to update you on some developments based on your suggestions. We understand that you are interested in dynamic filters and a focus on cross-product solutions.

Your suggested use cases for filtering, including filtering by custom fields, comments, issue links, project types or confluence content properties, amongst others, have been instrumental in shaping our approach. While there wasn’t a clear preference for a specific filter language or semantic, we’ve acknowledged the significance of flexibility in this context.

So, what’s next? We’re exploring at the moment different solutions based on your requirements. We’re primarily moving towards a cross-product solution for filtering. We’re confident in this direction, but we’re also realistic - if technical obstacles arise that we can’t get around, we’ll revisit our course of action. Our main goal is to deliver a dynamic filtering option that can be set by apps.

As we get closer to proposing a final solution, we’ll continue to keep you updated. Your feedback and participation are invaluable to us in creating a product that truly meets your needs. Therefore, we encourage you to stay alert for future communications as we continue this journey together.

Thank you for being an important part of our community. More updates will follow soon in this thread.

All the best,
Grażyna

[gkaszkur]

Hello,

Based on your initial feedback, we have explored different solutions, which involve utilizing external tools such as JsonPath or EventRuler. However, we aimed to have the flexibility to make changes and manage costs effectively. As a result, we opted to utilize the simplified version of Jira Expressions, which provides us with:

• Flexibility. It’s relatively easy for us to develop new features on this basis (for example, advanced event filtering using data from products)
• Simplicity. This approach makes it simple to create secure and efficient scripts for Forge apps.
• Performance. In our testing, no other solution performed significantly better than Jira Expressions.
• Familiarity. Jira Expressions are a proven and familiar solution that both we and app developers have a lot of experience with.

We’ll iteratively build a solution based on Jira Expressions. We’re setting the following milestones for ourselves:

1. Static filtering based on event payload via the app manifest (~Q4 2024).
2. Dynamic filtering configurable via the Forge API (~Q2 2025).

Technical details

Simplified diagram showing what event filtering will be working like:

We have opted for Jira Expressions with a limited set of operations (excluding data loading expressions in the initial iteration), with the potential to fully support them if the need arises in the long term.

We have introduced an extra step in event processing where events are compared against the filter specified in the app manifest. Any events that do not meet this filter criteria will be discarded before the Forge App function is executed. This process closely resembles the functionality of ignoreSelf.

For further information, refer to the simplified diagrams below illustrating the current and updated methods of event filtering.

Current Event to Forge App matching decision process

New Event to Forge App matching decision process

As you can notice, we have incorporated extra steps into the matching process, introducing a new filter expression outlined in this RFC. However, this addition has brought about some new challenges that we aim to resolve in a manner that best serves your needs.

The new filter will be declared with the new property of trigger module filter.expression. This would roughly look like this:

modules:
  trigger:
    - key: filtered-trigger
      function: trigger
      events:
        - avi:jira:created:issue
      filter: # new field under filter property
        expression: "event.issue.project.projectTypeKey=='servicedesk' && event.issue.fields.issuetype.name=='Support request'"
  function:
    - key: trigger
      handler: trigger.handler
app:
  runtime:
    name: nodejs20.x
  id: ari:cloud:ecosystem::app/<uuid>
permissions:
  scopes:
    - read:jira-work

Asks

If you have a use case that you believe is not addressed by this solution but should be, please share some details about this use case and the specific issues you have identified. Our goal is to ensure that the solution we develop caters to all Forge Apps.

Error handling

We also want to hear your thoughts on some more concrete problems. Expression evaluation may end with unexpected errors preventing the expression from returning a valid decision. Some of these situations can be caused by problems with the expression itself (we will validate expressions during app deployment, but some problems can be caught only during evaluation).

Therefore, we are interested in hearing your preferences in regards to error handling. We have several ideas in mind, and we would appreciate it if you could indicate which ones align with your needs:

• Option 1 - sending the event normally like there was no filtering
• Option 2 - sending the event with additional information about an error in the event payload
• Option 3 - dropping the event and informing about an error in the App developer console via logs
• Option 4 - choosing the preferred error-handling method by providing an additional property in the manifest

Trigger schema changes

One important point to note is that expressions are typically specific to the type of event. This may result in a situation where each event type is located within its own module, as illustrated in this example:

modules:
  trigger:
    - key: filtered-trigger-for-updated
      function: trigger
      events:
        - avi:jira:updated:issue
      filter:
        expression: "event.issue.fields.project.key == 'KD'"
    - key: filtered-trigger-for-created
      function: trigger
      events:
        - avi:jira:created:issue
      filter:
        expression: "event.issue.fields.project.key == 'SP'"

Do you anticipate any potential issues with the above example? Alternatively, do you have any recommendations on adjustments to the app manifest schema to maximize the benefits of Event Filtering for your specific scenarios?

We will keep this RFC open for the next two weeks to collect your feedback (October 2 - 16, 2024).

Thank you.

[klaussner]

Thank you for the detailed update, @gkaszkur!

Does this mean that filtering will only be available for Jira apps or are you planning to extend the expression language so that we can filter based on Confluence spaces, content, and entity properties?

[gkaszkur]

@klaussner

Does this mean that filtering will only be available for Jira apps or are you planning to extend the expression language so that we can filter based on Confluence spaces, content, and entity properties?

@klaussner Event filtering for Forge will be available for all types of product events. We will reuse the Jira Expression engine, however, the whole solution will be product-agnostic.

[SebastianHesse]

@gkaszkur Appreciate the intended solution for filtering, it looks promising!

Regarding your question what to do when problems arise, I’d prefer option 3) as this would be the first place where I’m looking for potential errors in my Forge app and was the first thing I had in mind before reading your suggestions.

I can understand why you suggested option 1) which follows the principle “a wrong expression is like having no expression” but I’m concerned this is not desired when you explicitly add such a filter expression. Whereas option 2) is the improved version of option 1) and definitely preferred instead of just sending the event without additional information. If option 4) is not too much overhead for you, then that might bring even more flexibility.

[SilvreLestang]

For the error handling, I think a mix of solution 1 and 3 is needed: sent the event to our app, but also write an error message to the log. This way the app still handle the event, which we don’t know if it should have been filtered or not, our code might ignore it depending on business rule. And we will see the errors in the logs, so we will know that something is wrong.