RFC-21: Real-Time Evaluations

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building shared understanding of a topic. It expresses a technical solution, but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Summary of Project:

Today, it takes about 3-4 hours for us to share evaluation (app trial) data with Atlassian Marketplace Partners. We have heard and understand this data is critical to partners for multiple reasons, including creating a smooth user onboarding experience. To address this issue, we plan to provide evaluation data in near real-time to the partners.

  • Publish : 17 August 2023
  • Discuss until: 4 September 2023
  • Resolve by: 18 September 2023


Atlassian’s Marketplace provides partners with reporting data which they use for running and scaling their business. This data includes details of the evaluations being done on their apps. As told by our partners, one of the use cases of this evaluation data is to provide onboarding details and run drip campaigns for their app to the customers evaluating it. Some of our partners also integrate this data with CRM systems to not only make faster decisions about their business but also provide a customised experience to their customers.


Providing data for evaluations takes about 3-4 hours to process and share the data with our partners. This is because we combine data from different internal sources that are processed at different rates (some slower than others causing latencies). Ultimately, this delay prevents partners from engaging with the customers in a timely manner after the license is created.

In addition to that, we are using a flat data structure. This implies that if the data in any field undergoes changes, we need to modify the data of all licenses showing that field. What this means is that the next time partners try to retrieve license data, they will be retrieving all the historical data that got modified because of that field. However, an update in some of the fields can act as noise and does not necessarily mean that the partners need to take any action. This task can consume a lot of time and can be a futile effort for our partners.

With our current way of exposing data, it is not possible for us to solve the above problems, particularly with near real-time data.

Impact of the problem on our partners

As a result of the delay in obtaining the data, partners have missed a few opportunities. One particular opportunity is to offer a seamless onboarding experience for their apps. The initial interaction with users can directly impact revenue by ensuring engagement, increasing awareness about the apps for customers, and thereby improving chances of conversion. Therefore, it is essential to engage and educate users while reducing the time required to deliver value to customers.

As quoted by one of our partners, a customer may install the app on a Friday at 2 p.m. and, because of the current delay in partners receiving the data, the customer might not receive the onboarding emails until Monday of the following week. In some cases, this is resulting in customers installing and then uninstalling the app due to not having the correct onboarding information.

Our objective is to increase the speed at which Partners receive evaluation data, which can help them reduce the time to first contact with customers who start a trial of their app. This has the potential to improve customer satisfaction and increase trial conversion rates. Additionally, we aim to enhance the way we distribute data and provide our partners with more flexibility in accessing the data.

Proposed Solution

Partners will be able to access evaluation data with improved latency (we are targeting a delay of less than 10 minutes compared to the current 3-4 hour timeframe).

The approach we are taking for this is to move to expose licenses as separate data entities. Data entities are defined as fundamental building blocks. In this context, we have Entitlement, Subscription, Contact and other entities.

Data will be exposed to the partners directly after a few minutes of the license being created. Partners can query and combine the data for these entities, based on their use case, via a graphQL API.

Please note - The images within the table below are presented separately due to technical constraints. Clicking on each image will open some parts of the table individually.

Disclaimer: The data entities and the fields associated with them are tentative and subject to changes. It does not represent the final version and is provided for illustrative purposes only.

To help you understand how our proposed solution will work, we have recorded a working demo for our partners.

Potential Impact of the Proposed Changes

Improved latency - The evaluation data will now be provided with improved latency, allowing partners to start their onboarding activities within 10 minutes of a customer beginning an evaluation. This is about a 95% reduction in the current delay. This improvement will provide an opportunity to educate and convert customers who may otherwise uninstall the app due to delays in obtaining onboarding details or other important information.

Improved data retrieval flexibility - Instead of utilizing an API to obtain all license fields which may not be necessary, partners now have the flexibility to retrieve data based on their specific requirements. Partners can choose to combine data from various entities to obtain only desirable data that would be most advantageous to them.

Moving to graphQL - In order to make data retrieval easier using entities, we will start exposing graphQL API. This is a change from the current way of reporting data via REST endpoints. We are moving towards graphQL because of the increased flexibility it can provide for retrieving data. Using GraphQL, partners can retrieve only the fields they need. It is possible to introduce new entities without any disruption. Partners can choose to use the entities at their own pace without their existing queries failing due to the introduction of new entities.

Once we release this functionality, partners will be provided with adequate documentation on all the entities, their significance, the fields that are a part of each entity, the relationships between the different entities which can be used to stitch them together and examples.

Future Iteration

For now, we plan to start with exposing APIs. In the future, in addition to APIs, we plan on introducing a push-based model so that the partners can receive the updates seamlessly.

Additionally, we will also be exploring how we can improve the quality of the data we provide to the partners.


While we would appreciate any reactions you have to this RFC (even if it’s simply giving it a supportive “Agree, no serious flaws”), we’re especially interested in learning more about:

  1. What do you think about using separated data entities to retrieve the desired data? We want to consider if partners find this kind of architecture flexible and useful
  2. Although the data entities provided above are not finalised, do you feel these data entities and the fields associated with them are sufficient for your reporting and campaign use cases?
  3. How do you feel about the problem we are trying to solve and any suggestions for us on this RFC?

We appreciate your time and expertise in providing feedback on these specific points outlined in the RFC. Your insights will play a significant role in shaping our decision-making process. When we close this RFC, we will inform you of our plans to incorporate your feedback by posting in this thread. At any point, if you feel that you would like to have a 1:1 with us to provide further feedback, ask additional questions or face trouble in understanding the RFC, feel free to raise a request to setup some time.


@GrushaBhalla thank you for this RFC!

With regard to this:

Does it mean that you expect Marketplace Partner to start polling the GraphQL API every minute to see if a new evaluation license is available? How is that going to work?

The link to the demo video seems to be protected, I need to log in to Loom and request access in order to see it.

@GrushaBhalla / @MalathiVangalapati can you please elaborate if the GraphQL API is going to replace the Marketplace API moving forward?

If not, moving this specific resource to a different API is only going to be a hassle for Marketplace Partners as it means that we are going to have to work with two different APIs providing the same data in different structures for no good reason.

If you’re doing this for Atlassian Marketplace Partners only, please don’t.

This feature looks good in terms of concept, but it’s screaming for either or both of:

  1. explicit confirmation that GraphQL subscriptions will be supported (and that whatever GraphQL gateway you are using will have the capacity to handle all of the vendors wanting to use this), and/or

  2. a complementary webhook API that pings one or more vendor URLs when new data is available.

These features should be a win-win and save you time in the long run, because it’ll save you from handling the load from tons of unnecessary polling. Doing only GraphQL subscriptions and not webhooks probably doesn’t work though, since the implementation complexity for GraphQL subscriptions is relatively high and it creates a high burden for vendors to redo their entire tooling.

I also really like the ability to see the decomposed data structure. It will be important to ensure the immutability of as many of the ID or ID-type fields as possible, across all of the data structures, and to explicitly indicate in the documentation which fields are permanent ids and which can change. This is because many of us are syncing these to our own databases and we need to know whether to update or create new records.


Perhaps if you put all this detail somewhere similar to /rest/api/settings/systemInfo or indeed into the webhook body, then any install webhook could be used to send the onboarding emails in real time.


We would like to express our gratitude to all of you for taking the time to read the RFC and provide feedback. We will respond to your comments shortly. Additionally, I would like to apologize for any inconvenience caused by the demo video being inaccessible. We have resolved the issue, and all partners should now be able to easily watch the video.

I know this is some unique selling point for GraphQL, but in this case, it really does not make any sense. We’re processing license information in the background. There are no constraints on data transfer in terms of speed or size, as there would be on a front-end application.

If any, switching to a different data structure compared to the already existing license report output basically means I can’t re-use code and only makes my effort to work with Marketplace API more complex, both in terms of initial development as well as maintenance.

Be cool if we could get the email of the person doing the actual installation. Onboarding drip campaigns are mostly useless when your only accessible email is an admin/technical contact which is often either a CEO/CFO who has no idea on the context.


@nathanwaters +1 on this. Of course the technical contact is also normally out of date information or has already left the company, so it’s even worse than no context.

1 Like

I think what we’d be looking for is who the installer thinks is the main contact for the app. Especially in an Enterprise setting. But that’s def outside of this RFC (but if anyone at Atlassian wants to dig into it… :slight_smile: )

Now back to the RFC at hand:
+1 on the limiting of the fields. (Depending on our processing utilities - we don’t want to extract more info than we need to).

Will it be possible for us to query in graphql so that we only get certain items based on dates? (Ie we’d love to make sure that we don’t get email adresses for folks that haven’t been a subscriber in X days).

It would also be great if there was a proper webhook/subscription registration path so we don’t have to poll for things every 5 minutes (or the last time we got a successful fetch).

Neat to see Atlassian thinking down this track. +1 to thinking about pairing this solution with webhooks.

  1. In light of that not being proposed at the moment, what would be a reasonable rate of querying the data that wouldn’t overload the system on your end?

  2. Also for the exisiting license and transaction export, what is a reasonable rate of querying that might get new data. It sounds like 3-4 hour timeframe at present?

  3. With this new GQL endpoint - is it a completely separate API to the license and transaction export. Will that be affected by this new RFC?

I share the concern raised about two different sources of truth coming from Atlassian in two different schemas.

1 Like

Hi everyone!

We would like to express our gratitude to all of you for taking the time to read this RFC and providing your valuable feedback. We have tried to collate the questions as per some of the emerging themes:

  • Push vs. Pull Mechanism: We will assess this further and share it with this group. Regardless, we want to ensure that even with a pull mechanism, there would not be any concurrency issues.
  • Having the appropriate customer contact info: We understand the importance of right customer contact for our partners. We are already in talks with our compliance and legal teams about this and will share an update soon.
  • Deprecation of current APIs: Some partners have inquired whether the proposed solution will replace the current Marketplace Licenses APIs in the future. In the immediate future, we have no plans to replace our existing APIs with the proposed method of providing data. It’s important to note that these two functionalities will cater to different use cases. Our proposed solution aims to provide a subset of fields in license data faster, which will only be applicable to evaluation data. We have designed this solution to help our partners provide a faster and smoother onboarding experience to their customers. Having said that, we will keep our partners in the loop for how our APIs will evolve going ahead.
  • Documentation: Once we go live, we will provide extensive documentation to assist partners in onboarding to this new method of retrieving evaluation data. We will also incorporate the immutability of fields as a detail, as requested by some of you.

@remie :

We acknowledge that transitioning from REST to GraphQL may present a challenge for our partners. However, we have decided to make this shift in order to optimize certain aspects of data exposure through proposed data entities.

  • Single API call for accessing multiple entities:
    • With graphQL only a single API call is needed even if you want to access different entities according to your use case
    • With REST, multiple API calls are required, one for each entity
  • Addition of new fields:
    • With graphQL, in case of addition of new fields, existing queries will still continue to work
    • With REST, in case of the addition of new fields, partners might need to update their way of reading the response depending on the way they consume data

Partners can also optimise their processing and data handling capabilities by getting only the data they need.

@JessicaFosler :

  • For our current APIs, we provide partner evaluation information with a delay of 3-4 hours. It is recommended to use a similar polling frequency to access the latest information.
  • Though it will provide data for a subset of fields, this is a completely separate API from the current licenses and transaction APIs. The RFC does not affect those APIs.
  • The proposed solution will provide a subset of the data provided by licenses and transaction APIs with a much lower delay. The idea is to provide partners with limited but relevant fields in a faster manner and enable them to send onboarding emails faster.

Additionally, we kindly request the community for their input on:

  • The value of real-time evaluations
  • The effectiveness of it for the current proposed solution

We appreciate all the feedback you are giving us.

Thank you!

1 Like

@GrushaBhalla, thank you for your response!

Apart from acknowledge that you are imposing additional burden on partners by not only moving to GraphQL but also creating a split brain situation in which the same data will be available in two different API’s, can you please also provide us with the reasoning behind moving to GraphQL, e.g.:

What problem are you trying to solve?
Who requested this problem to be solved?

You are now only offering solutions as the reasoning (single API call, field selection) but you are not telling us why this is a problem to begin with that needs to be solved and why this problem outweighs the burden of dealing with split brain.

The reason I asked if you will be migrating is because the answer to my question could also be that GraphQL is going to be the new standard within Atlassian. But if this is not the case, it would be great to know why we are being asked to spent time dealing with Yet Another API.



1 Like

@GrushaBhalla, I’d love to hear why this information couldn’t also be sent via webhooks on install. This would mean actual real time information being sent that could then help with onboarding email in real real-time, not a little while later.


@GrushaBhalla , we would be very thankful if you could also extend the License API for cloud apps with the license end date. Currently, we get this information only from the Marketplace API.

Additionally, you should consider fixing how you handle the apps lifecycle internally so that you fix these open issues:

Hi @remie

Thank you so much for your feedback and questions. I ll try to break the response down according to your questions.

What problem are we trying to solve?

The issue we are addressing is the prompt delivery of evaluations. Currently, we provide our partners with this information after a delay of 3-4 hours. This delay results in a slow and inefficient onboarding process. To reduce this delay, we will be presenting the data in the form of entities, as suggested in our solution. We want to emphasize that our goal is not to transition from REST to GraphQL. Instead, GraphQL is a more efficient way of consuming data entities proposed in our solution, as explained in our previous response. Therefore, we will be using GraphQL in this solution instead of REST.

Who requested this problem to be solved?

Making evaluation data available in near real-time has been a partner request. We had also consulted with a few partners before proposing the solution about the necessity for real-time evaluations.

Hi @david

Thank you for your feedback. We are evaluating the possibility of implementing webhooks as well. However, launching without webhooks will help us launch faster.

Hi @GrushaBhalla,

I’m not sure if you are deliberately avoiding answering my question, but for the slim change I wasn’t clear enough I will repeat myself, because your answer does not match my question.

What problem are you trying to solve BY USING GRAPHQL OVER REST?


I’m not asking questions about having evaluations data more quickly. We all understand that this is a very important change that was requested by the partner community.

But for a yet undisclosed reason, Atlassian has decided to move away from the already existing REST API and change to GraphQL without a clear reason why Atlassian is doing this.

Please provide us with the reasons for the move to GraphQL, as this is not clear from the RFC itself. Atlassian can also reach the same goals of this RFC with REST API.

Please address these questions properly without repeating the same meaningless answers.

1 Like

Bump @GrushaBhalla

CC: @ibuchanan

Hi @remie !

Thank you for providing additional details about your question.
I apologize for the delay in responding to your comment. We are currently evaluating your feedback as well as all previous feedback for this RFC, which is causing a delay. We request you to wait for a response from us by this Friday (September 22, 2023).
Once again, apologies for the delay and thank you so much for your feedback!