Addressing issues with Marketplace reporting and APIs

We want to acknowledge that this year, Marketplace reporting has not been meeting the standards of reliability you should expect as our partners. We know this impacts your teams, your ability to accurately assess your business, and often your ability to support customers, and we apologize it has taken this long to address the issue. We are committed to improving the experience and being more transparent with you about this process.

Last September, Atlassian transitioned from an outdated data platform to a new one, a change that was intended to improve ongoing reporting issues. Over the subsequent months, we realized that replacing the old platform did not address the root cause of the reporting issues. The data pipeline feeding the platform relies on 10+ dependencies with multiple potential sources of data pipeline failure. These multiple sources have contributed to roughly 75% of incidents over the past year.

To address the resulting breakages, inconsistencies, duplications, and general unreliability, we’ve spent the past several months reviewing and reworking our data pipelines and operational processes to reduce dependencies, simplify data acquisition, implement checks at each step, and streamline internal support.

Specifically, we plan to update four points of failure in our data pipeline related to pulling data from our Finance system, moving to a data model with improved latency (see below).

These changes will resolve many causes of delayed financial data and significantly improve data integrity.

While these updates will result in improvements, they will not solve all of the issues Marketplace Partners are experiencing with reporting and APIs. Updating the data pipeline lays a foundation that will allow us to deliver the experience you expect and deserve going forward. Following the release of this new pipeline, we will continue to work on addressing different root causes behind the remaining 25% of incidents we’ve had.

This initial phase of updates to the data pipeline will lead to the following improvements for Marketplace reporting:

  • Reduced latency and delays:
    • 90% of partial data delays (delays < 72 hours) will be resolved
    • 70% of full data delays ( > 72 hours) will be resolved
    • Expected improvement: most 12-36 hr delays will be reduced to less than 6 hours
  • Data integrity improvements:
    • 99% of data duplication incidents will be addressed by new data checks
    • 99% of missing data incidents will be addressed by new data checks

These data pipeline updates will not solve:

We will continue to work on addressing the following, but do not have detailed solutions or timelines at this time.

  • Latency and delays:
    • 10% of partial data delays (<72 hours) will not be fixed due to a continued dependency
    • Data delays at the end of the month - month-end data delays are caused by limitations on number of concurrent updates certain source systems are currently able to process. This data pipeline project will not address this concern. At this time we recommend waiting 2 days for month-end data.
  • API availability :
    • 3rd party tools have caused issues in the past. We are working closely with our vendors where we have seen problems but more work is still needed.

Some data may look different due to these pipeline changes

While there are no changes in the API contracts or in the data model, Marketplace Partners may see some changes in the data due to the updated data pipelines, including:

  1. Some Partners might see a change in the contact details on licenses - Null, inaccurate (ex - test) previously, and accurate contact info once the changes are implemented due to the improved pipeline. Most recently modified contact fields will show up if there are multiple contacts associated with a customer
  2. Parent product name, parent product edition, parent product billing cycle in parent insights may have revised values in them, because of improved data quality in the new pipeline. In the new pipeline, data is coming from the source, whereas in the legacy pipeline this data was derived through a calculation. In the legacy pipeline, we had to derive the parent based on formula logic where if a customer’s site has multiple Jira instances, the edition is first determined based on user count, and if the user counts are the same, then prioritized as follows: Jira Software > Jira Service Management > Jira Work Management.
  3. The ‘updated date’ field will reflect any update to licenses and transactions, addressing limitations we’ve had in the current pipeline (see example).

We are currently testing the new pipeline internally to identify other differences you might experience in reporting once the new pipeline is launched. If additional differences are identified, we provide those details prior to switching to the new pipeline so you know what to expect and have adequate time to prepare.

We plan to release the pipeline changes before Marketplace shifts to a new cloud billing and subscription management system. We will not move apps onto the new billing and subscription management system until the new pipeline is in place and achieving the expected outcomes listed above. Please note: this means that the billing and subscription management system roadmap (linked above) is subject to change, as it is dependent on our new reporting pipeline.

In order to better understand the impact of the new pipeline in the context of Marketplace Partner reporting setups, we will first release to a small group (Early Access Period / EAP) who will provide us with extensive feedback so we can adequately communicate and prepare you.

Our initial plan was to start a 4-week EAP in July. However, during our internal testing and EAP preparation, we made the difficult decision to do another round of diligence with senior engineering management across all contributing teams reviewing the technical changes, monitoring, support and incident management plans for the new pipeline before proceeding to EAP.

We will be holding a live Product Team Talk Time session on this topic in the coming weeks and giving monthly updates here in the Partner Portal. We will share more details in the next updates.

With these changes, we hope to provide you with reliable, accurate, and up-to-date data so you can reduce time and resources spent troubleshooting and re-running data loads and focus on running your business and bringing value to our shared customers.

Again, we apologize for the experience we’ve provided you over the past year and will work to regain your trust by delivering the kind of reliable experience you deserve.

Thank you for continuing to provide feedback. We want to simplify and improve how Atlassian developers and Marketplace Partners find the information and tools to succeed. As part of these ongoing efforts to improve Marketplace Platform Stability and Reliability, we will take active steps to incorporate more of your feedback to bring you the experience you need and expect as Atlassian Marketplace Partners.

Reminder: Atlassian Marketplace is migrating to a new cloud billing and subscription management system in the coming months, which will have some temporary and some permanent impacts on Marketplace reporting in addition to the changes outlined in this post. Please be sure to review Purchasing System Update Marketplace Impacts to prepare.

7 Likes

Can you elaborate on these classifications? It seems like Atlassian is conflating duration of delay with completeness of the data?

We’re not interested in participating in the EAP, but would be happy if Atlassian used our code at GitHub - Atlas-Authority/marketing-automation pointed at new and old data pipelines to help ensure the changes work as expected.

1 Like

Hi @MalathiVangalapati!
Thanks for this.

Couple of questions.

  1. Can you speak to what happens when there is an outage? What are the escalation paths, SLAs, and remediation follows up? Is any of that changing?
  2. In all of this - there’s nothing really mentioned about how detection is done. Can you speak to that? What are you doing to improve in this area?
  3. For the apps that Atlassian has acquired and still sell on the Atlassian Marketplace - at what point in this rollout do you think that the Atlassian Finance team, Atlassian Marketing and Atlassian Compliance team would feel comfortable with relying on the Atlassian Marketplace reporting APIs as being their only source of truth? Is this at the EAP, right after the EAP, right after all of this being rolled out everywhere and so forth?

Thanks,

/Daniel

@MalathiVangalapati I’d like to go even a bit further than @danielwester and would say that we’re getting at a point in which it is time for Atlassian to face the music.

There has to be some sort of accountability with regard to the stability & reliability of the service and the integrity of the data. I’m sure you all have (personal or team) OKRs that are related to MPAC performance, but as a customer A) I don’t know about these, and more importantly B) I don’t want individuals to be “punished”, I want Atlassian to feel the pain as a company.

Which means that in addition to having SLAs or SLOs, I think these should also be incentivised. This has been mentioned before several times on this forum. There should be a financial impact if you want to restore trust to this community. Otherwise, these are just empty words.

To make it more concrete: let’s say no revenue share for Atlassian for every day that there is an outage. And to make it clear that I don’t care about the money, but more about the principle, I’m more than happy to donate that revshare to a charity. Heck, I’ll even do it in Atlassian’ name.

But we have to get to a situation in which the pain is actively felt by Atlassian, and not just the Atlassian Marketplace Partner community.

EDIT: I can imagine that this is not your call to make. If that is the case, I would like to add an etiquette rule to this forum: please tag the person in your org chart that those have the authority to make this decision instead of just ignoring this or bullshitting your way out of it.

1 Like

Hi @MalathiVangalapati ,
Thank you for the update. I really appreciate that you took the time and think you are spending much effort to improve the marketplace.
My feeling is that these improvements are “engineering driven”, because many of the outages seem to be caused by technical failures.

Can we also get the viewpoint of Atlassian accounting/finance on the impact on the financial reporting? Maybe this is out of your remit, but you may be able to tag the right persons.
Thanks.

Hi @remie Atlassian is discussing this along with other means of ensuring we are accountable to the right standards of reliability and stability in our platforms. We don’t have specifics or any decisions at this time since there are multiple teams involved in this decision, but @AmarSingh and I will be the main points of contact for partners within Atlassian leadership as this progresses.

Hi @marc The APIs serve up data from a copy of the source systems, so the latency and integrity issues we are experiencing are the result of data getting stuck in different places within the pipeline itself. The issues are not impacting the source systems/data which is why this is an engineering-driven effort and not an accounting/finance one. However, we understand that what is an engineering effort for us has an accounting/finance impact on our partners.

However, please let us know if there’s a specific detail you believe may still need more information or response.

2 Likes

Hi @BorisBerenberg , Thanks for sharing your code, this is helpful. We are leaning into our Marketplace Partner community to gather more such inputs to ensure we incorporate these checks in addition to our current checks.

By partial delays, we are referring to instances where a transaction or a license is available in the reports/API but the associated information like contact or partner name is delayed.This data was usually picked up in the subsequent runs of the ETL jobs and showed up in the reports/API with a delay of 24-72 hours

By complete delays, we are referring to instances where the entire record of transaction or license is missing . In the past it took over 72 hours for this data to show up in the reports/API.

With the enhancements in new pipeline, we will be addressing 90% of the issues leading to partial delays and about 70% issues leading to complete delays.

In addition to fixing these issues, we are improving the latency of transactions and licenses to less than 6hours

1 Like

Hi @danielwester ,

  1. We will be launching a new incident category when we migrate Developer and Marketplace Support July 25. Partners are encouraged to use that over the current CDAC category we’ve had in place up until now and we will be deprecating the CDAC incident category within the months following the launch of the new support experience. As for additional changes to SLAs, remediation follow ups we are still working through necessary changes related to incidents in general so will provide more details as we have them ready.

  2. We will do a live session prior to launching the EAP where we will go into detail on our detection mechanisms. In the meantime, if you or others in the Marketplace Partner community have built your own checks we would appreciate you sharing details so we can let you know if we believe we’ve replicated something ourselves or help us determine if there are areas we’ve missed.

  3. We understand why this is an important benchmark. We can be confident that Marketplace reporting meets the right standard for our Marketplace Partners’ finance teams when we feel it would be reliable enough for our own reporting purposes. We’re going to need a bit more time to define this benchmark and timeline, but we are tracking your question internally to make sure we address it in a future update so it doesn’t get lost in this thread.

1 Like

Hi @amardesich ,
Thank you for the reply, this is really helpful. If I understand your description correctly, Atlassian has a working financial records system. That is important to us vendors, as it can serve as a source of truth.

1 Like

Today’s outage was detected by us nearly 4 hours earlier than reported on Atlassian statuspage and we only check once every 4 hours, so may have been down for 4 hours at that point.

We will be launching a new incident category when we migrate Developer and Marketplace Support July 25 . Partners are encouraged to use that over the current CDAC category […]

I suppose the new incident category is “Report a Marketplace Incident”. Is that correct?

Is it possible to convert an existing “old style” incident report into the new category? I’m asking this because our existing report doesn’t seem to get any attention.

Hi everyone, thanks for your questions and feedback so far. As mentioned in this post, we are providing monthly updates on our progress on the partner portal. Our first monthly update is now live here.