RFC-74 : Transactions for Forge Storage

General Request for Comments (RFC)
, , ,
1

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building shared understanding of a topic. It expresses a technical solution, but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Project Summary

This RFC proposes the implementation of transactional operations in Forge Key Value Store and Custom Entity store to address the need for atomicity in data operations, which is a significant pain point. The solution aims to provide a reliable way to perform multiple operations in a single request, ensuring data integrity and consistency.

Publish: 22 November 2024

Discuss: 06 December 2024

Resolve: 20 December 2024

Problem

Developers face challenges in ensuring data integrity when performing multiple operations due to the absence of transactional support in Forge. This limitation leads to issues like race conditions and data inconsistency, particularly in complex use cases.

Proposed Solution

  • The proposed solution involves implementing atomic operations on top of Key Value Store and Custom Entities. This will allow developers to perform atomic operations, ensuring that all operations succeed or fail together.
  • The solution will further include support for conditions on top of Entities, enhancing the ability to enforce business logic and data constraints.

Asks

While we would appreciate any reactions to this RFC, we are especially interested in learning more about:

  1. What use cases can you unlock with atomic operations on Key Value Store and Custom Entities, like ensuring all updates succeed together in a batch process?
  2. Conditions(for example equal to) will likely be supported only on the Entity Store, not the Key Value Store, due to the need for typed data sets. Would this possible limitation block any of your proposed use cases?
  3. Are you using transactions with other data stores because Forge doesn’t support them yet? If yes, do you use conditions like “equal to” for data validation?
  4. How many operations do you need in one transaction to support your task, such as updating multiple records at once?
  5. Is the order of operations within a transaction important for your use case? If yes, please help us understand the use case.

We welcome feedback and suggestions to refine the proposed solution and ensure it meets the needs of our developer community.

8 Likes
2

Thank you for moving this important topic forward. Missing transactional access has always been a pain point when using the current Forge storage capabilities.

I’d like to explain my main use case with Templating.app. As the first Jira Forge app on the marketplace, it uses the KV Storage API combined with various hacks to achieve something similar to atomicity.

The simplified use case is as follows: A site can have thousands of Issue Templates. Each template is stored as a value in the KV Storage. Users need to search through the names of all these templates, which is currently not feasible due to the limited query capabilities and the query limit of 20 (recently increased to 100). To work around this, the app maintains an additional KV pair that acts as a ‘search index’ (I shouldn’t call it like that). This index is essentially a list of all template names (and other searchable attributes). Every time a template is created, updated, or deleted, the search index needs to be updated and kept in sync with the actual KV pairs.

The problem arises when multiple CRUD operations occur within the same timeframe—ultimately, the last write to the ‘search index’ wins. To mitigate this, I implemented a custom lock/release mechanism. While functional, it is slow.

What I would need from the proposed API extension is a (pessimistic) lock mechanism that ensures reliable parallel writes from different Lambda invocations. Specifically, if a second invocation attempts to write to the same KV pair, it should wait until the lock is released. If the lock isn’t released within a specified timeframe, it should fail (along with other associated write operations).

An example of how this API could look is illustrated in Firestore: Transactions and batched writes  |  Firestore  |  Firebase

For my use case, the order of operations isn’t critical, the number of operations per CRUD action ranges from 10 to 15.

I appreciate that Atlassian plans to improve the storage capabilities. However, I’d like to express some concern about the introduction of three separate storage mechanisms—Forge Storage KV, Forge Storage Complex Entity, and Forge SQL—each apparently with different capabilities. Most of our apps rely on the KV version, and I fear it could be abandoned or deprecated in the future. Given the nature of Forge, migration could be especially challenging without direct access to the data.

Clear communication regarding Atlassian’s long-term vision for Forge storage solutions would be great. Especially, if the KV becomes deprecated at some point, I’d expect an migration process supported by Atlassian.

Best,
Julian

13 Likes
3

Hello @SunandanGokhroo,

Thanks for bringing up this important topic - the lack of this has also bothered us from the get go some years ago!

Our present main concern are mainly race conditions on write, for example when voting and two people vote at the same time, we want to add both votes to an array and update the value. So this is basically one transaction (we built our system due to the limitations that way), but with a couple like 2, 3 transactions, that would make our life easier.

But the before mentioned seems to indicate we need some update field in our data, or that the action can run in a lambda that always gets the current state and can produce a valid next one. If this can be only done with Custom Entities, we face a again a migration problem: how do we get data from one store to the other?

Best
Eckhard

2 Likes
4

This would be useful for our app if it can act like Forge cache where we can check if a value is set before setting a value. We have a postfunction that could be launched multiple times concurrently and an issue needs to be created exactly once no matter how many are run. Our users are trained to move the tickets one at a time to get around this. Full transaction support isn’t necessary for our use case: a queue that could be configured to run 1 at a time, Forge Cache, and concurrency mechanisms from the new SQL database could also work.

3 Likes
5

Hi @JulianWolf thank you so much for the detailed reply and helping us understand your use case better. This is invaluable to us in ensuring we are able to build something to serve the community better. Regarding the concern raised about multiple storage solutions with different capabilities, I want to reassure you that we are actively invested in all three and our intent is to only ensure that we are able to provide the required flexibility and options to unblock the community.