TL;DR: In this update about how we’re stabilizing the Atlassian Marketplace reporting and APIs, we share information about our upcoming plans to transition from Early Access Program to GA rollout of the new data pipeline.
Timing: The transition to the new data pipeline will occur as part of a phased rollout starting 13 October 2022 .
We’d like to ask all the partners to take into account how important this change is, and thank partners for going through this transition with us. A 100% transition to this new data pipeline is critical for not only partner’s success in their day-to-day business operations, but also a prerequisite to continue with the rollout of the new Atlassian billing engine for all customers.
We ask for your help and attention throughout the coming weeks as we work to transition every partner to the new data pipeline. Read on for more detail on rollout timing, steps to prepare and methods to raise feedback and questions.
If you foresee any problems with transitioning to the new data pipeline for your company, please let us know immediately in the the comments of this post.
Since our last update: Findings from Early Access Program (EAP)
Since our September update on the Partner Portal, we’ve introduced the following to ensure partners are prepared for the rollout of the new Marketplace data pipeline:
- Product Team Talk Time session: On 14 September, 2022, we shared how Atlassian is reworking our data pipelines and operational processes to reduce dependencies, simplify data acquisition, implement checks, and streamline internal support.
- Early Access Program (EAP) began: On 21 September, 2022, we kicked off a high-touch EAP for the new data pipeline with 14 partners. During this time, we worked with Marketplace Partner volunteers to validate data pipeline improvements, address feedback, and resolve issues.
We have also performed two validation steps for additional data quality assurance with the new pipeline. These steps were introduced to ensure that the quality of data we provide partners is on par with what Atlassian teams use internally:
-
Reconciliation of new Marketplace pipeline against Atlassian’s internal reporting pipeline: To ensure that the data quality meets Atlassian’s own internal standards for reliability, we validated the new data pipeline against data consumed by Atlassian teams for baselining and forecasting revenue. In this exercise, we found that evaluations differ by ~0.4% and sales differ by ~0.2%. Our learnings on the discrepancies and plans to resolve the remaining gaps are detailed in the expand below. This has led to a new process and set of reports that will flag discrepancies between Atlassian’s data pipeline and the one that provides data to Marketplace Partners, meaning Atlassian will be proactively monitoring for issues and flagging them before partners notice them.
-
Reconciliation of the new Marketplace pipeline against the old pipeline, an internal dashboard for monitoring created: Another reconciliation process compared data between the old Marketplace pipeline and the new one. An internal Atlassian reconciliation dashboard was created to proactively identify any issues in the transition to the new pipeline. We are monitoring this dashboard to make sure the new pipeline data is as close as possible to current data while also improving accuracy. This page includes the findings from this process (see “ Differences you may notice in the new pipeline” below).
Detailed outcomes of validating data between Atlassian reporting pipeline and new pipeline
Sales ($ amount) and evaluations (trials) are found to be accurate in new data pipeline; Any discrepancy in the new data pipeline is either due to known or expected reasons.
- Evaluations: The new data pipeline, similar to the old data pipeline, includes test licenses, while internal Atlassian reporting does not. Hence, the monthly difference in the evaluations count is ~0.4%.
-
Sales: There is a monthly discrepancy of ~0.2%. This is due to two reasons:
-
Reason 1: Certain large invoices (>250KB) are missing in the new data pipeline.
- Cause: Events not landing in Atlassian’s event bus due to size limitations. We are working on a solution for this in subsequent phases.
- Impact: The large invoice problem is something that exists today in the old pipeline as well. We’ve noticed that it only happens occasionally, for example, since 1/1/2019 it has only happened 8 times. If it does occur, partners still receive payment for the transactions, but no invoice is generated.
-
Reason 2: Difference in invoice date between the new data pipeline and internal Atlassian reporting
- Cause: Difference in timezone between new data pipeline (UTC) and internal Atlassian reporting pipeline (CST)
-
Reason 1: Certain large invoices (>250KB) are missing in the new data pipeline.
What we are working on now: GA rollout begins this week
We’re excited to share that the new data pipeline will soon be slowly moving out of the Early Access Program and into General Availability through a phased rollout. This means you will soon experience improved data latency, integrity, and service up-time on the Marketplace Portal.
We’d like to ask all the partners to take into account how important this change is. A 100% transition to this new data pipeline is not only critical for success in day-to-day business operations, but for continued rollout of the new Atlassian cloud billing engine for all customers and apps. The new billing engine is a key prerequisite for a number of cloud billing improvements that will benefit Marketplace Partners, including editions for cloud apps.
We ask for your help and attention during the coming weeks. We acknowledge that this transition may be a challenge as it impacts such a fundamental piece of your business, and we are prepared to support you. Steps to prepare and methods to raise feedback and questions are listed below.
Timing of rollout
This phased rollout will begin on with a small group of partners. We will share more details around the timeline for the rollout and start dates for each phase by 17th of October.
We chose to take a phased approach to this release so that we can ensure that each of our partners experience minimal disruption, can receive proper support and attention and that any remaining issues can be caught and mitigated quickly. We also have a contingency plan for a rollback plan if needed (see details in the expand below).
Conditions for rollback of GA pipeline
We have prepared and thoroughly tested the pipeline and improved our alerts, monitoring processes and dashboards. We can’t rule out finding new issues during rollout, however, with the use of feature flags, we will be able to restore functionality if and when we encounter significant issues. We would roll back the changes if:
- There are major data issues affecting multiple partners
- Financial data discrepancies
- Contact data discrepancies
- PII data issues
- Security or privacy issues
We are aware of some data differences, as communicated in “Differences you may notice in the new pipeline” below.
How we will notify partners which phase they are in for the rollout
If you belong to the first two phases (that will complete the transition to the new pipeline by October 31st) your TPM will notify you via email, and the email will be sent to the technical contact and the primary admin contact(s) in your organization. If you belong to a subsequent phase after October 31st, we will add your vendor name to a comment posted on the update on the Partner Portal to notify you in advance that you will be up next to be transitioned to the new pipeline. Please make sure that you have subscribed to the post linked here in the Partner Portal so that you do not miss these future updates.
Required actions for partners to prepare for rollout
Given the criticality of the new pipeline rollout and the different ways partners consume this data today, please:
- Familiarize yourself with the differences you may notice in the new pipeline
- Complete the following steps to prepare in the event of a rollback (if this happens, we will share a detailed plan on the steps to take)
If you have comments or questions, please comment on this post so we can get back to you. If you observe any issues in your data during the rollout period, please raise them by submitting a ticket via this service desk.
| Current data pipeline usage by partners | Recommended steps for partners to prepare for new pipeline (please review the changes to reporting behavior and data detailed below): |
|---|---|
| Partners fetching lifetime data every day (i.e. partners consuming data using lifetime export APIs everyday and maybe having bespoke automations, integrations configured on the data (CRM, Marketing etc.)) | Action Required: Take backup of the pre-rollout data pull (one day before the roll-out) → Pause the automations like email triggers, integrations (CRM, marketing) for a day or two, if you are using any → Pull fresh data after the roll out communicated by Marketplace |
| Partners fetching delta data every day (i.e. partners consuming new data using export APIs every day, which may have bespoke automations configured on the data) | Action Required: Take backup of the pre-rollout data pull → Pause the automations like email triggers for a couple of days → Delete data from rollout date onwards → Pull fresh data from rollout date |
| Partners consuming reports via Marketplace UI only at https://marketplace.atlassian.com/manage/vendors/$partner-id/reporting/ | No Action Needed: There is no need do take special measures in this case. In case of roll-back, changes would be reflected immediately in the marketplace UI. If you want, you may take screenshots of the UI to compare old vs, new data |
What can Marketplace Partners expect from us next?
Over the course of the next month, we will be communicating which partners will be transitioned to the new pipeline. In our November update, we will share how the transition to the new pipeline is progressing, along with the next steps to ensure all partners success during general availability.
Transition to the new data pipeline is a critical step in enabling apps on new billing platform. For more information on navigating changes to Atlassian’s billing engine please see this post.
Differences you may notice in the new pipeline
The average data availability latency is expected to decrease to ~6 hours. i.e. From the creation of a license and transaction, it used to take ~24hrs to reflect in reports. The expected improvement depends on the time taken for internal jobs running on data systems, with an average of ~75% improvement:
GET Licenses API (GET /rest/2/vendors/{vendorId}/reporting/licenses) and Export Licenses API (GET /rest/2/vendors/{vendorId}/reporting/licenses/export)
- We can expect an average of ~3hrs latency, best being ~1hr and worst being ~33hrs.
GET Transactions API (GET /rest/2/vendors/{vendorId}/reporting/sales/transactions ) and Export Transactions API (GET /rest/2/vendors/{vendorId}/reporting/sales/transactions/export )
- We can expect an average of ~6hrs latency, best being ~ 3hrs and worst being ~33hrs.
Cloud churn, conversions, renewals, license events APIs
- We can expect an average of ~6hrs latency
| Change | Details |
|---|---|
| New update Revenue fields are rounded off to 2 decimal points | There is inconsistency in the revenue fields such as purchasePrice,vendorAmount,partnerDiscountAmount etc in the old pipeline |
| Get transactions API and Exports Json doesn’t round off the values in these fields, but the values are rounded off to 2 decimal points in csv and reports UI. | |
| To maintain consistency with remittance reports, we have rounded off all the revenue fields to 2 decimals. | |
| No change to the APIs | The API signatures and fields remain the same in the reports and we have fixed the data anomalies, as reported previously by our partners. |
| Partners should anticipate that some data may change, including potentially primary identifiers like emails and names. | |
| Frequent data refresh | The new pipeline will be refreshed more frequently and you would have the latest data in comparison to old pipeline. Licenses are available in ~3 hours and transactions in ~6 hours from the time of creation. |
| License and transactions may not be in sync always | As licenses are available in 3 hours and transactions in 6 hours since creation, some licenses will be available early and corresponding transactions might show up in subsequent data pulls. This behavior not new, and occurs today in the old pipeline. |
| Transactions show point in time (when the transaction was created) contact details whereas licenses show latest contact details. This means that contact details for license need not be in sync with latest |
|
| Accurate updated date (lastUpdated) field | The ‘updated date (lastUpdated)’ field in the licenses and transactions API will reflect any update to licenses and transactions (that were previously missed), addressing limitations we’ve had in the old pipeline. |
| During the migration, most of the data will appear as having been updated, even though there may not be a change in the data for the existing license. | |
| Updates to historic |
Historic licenses may be updated because it reflects the latest information of a license. A few examples of fields that can get updated are: license state, contacts, etc. |
| Ledger fields ($ impact fields ex - purchase price, vendor amount) on |
|
| Updates to inactive licenses | Inactive licenses will not be updated except for handling RTBF (Right to be Forgotten) in billing and technical contact details, app name, vendor ID (acquisitions/mergers). |
| - When an app name changes from App1 for Jira to App2 for Jira, historical licenses will also start reflecting App2 for Jira | |
| - When an app of partner A is acquired by a partner B and if the consolidation (transfer) of data was initiated by the partner B, all the historic licenses of the partner A will be updated and attached to partner B | |
| Differences in Contacts | Some Partners might see a change in the contact details (ex. technicalContactName, billingContactEmail ) on licenses - Null, inaccurate (ex. test) previously, and accurate contact info once the changes are implemented due to the improved pipeline. Most recently modified contact fields will show up if there are multiple contacts associated with a customer. |
| With the new pipeline, we have moved to a new source for contacts called Public Data registry . This is used Atlassian wide as a the source of truth for contacts. With this migration, there can be a few data differences: | |
| - The order of first name and last name might change (ex. John Doe might show up as Doe John and vice versa) | |
| - There could be addition or removal of accented characters (ex. John Doe might show up as Jòhn Doè and vice versa) | |
| - There could be a additional or removal of localised characters (ex. John Doe might show up as 张三 and vice versa) | |
| Change in the license, |
In the old pipeline, when an acquisition or merger happened, historical licenses may still remain to be associated with old vendor/partner. In the new pipeline, historical licenses would be transferred to new vendor from day of acquisition. |
| In the current pipeline, a few Atlassian licenses were showing up as a result of a bug in reports which ideally should be excluded. In the new pipeline, we are excluding such internal Atlassian licenses. There is an ask from a few partners to excuse licenses of Bugcrowd (licenses created as part of Bug Bounty program), however we are still evaluating partners’ interest in this regard before we exclude this. | |
| Until now, churn, conversion and renewal trends and counts included internal Atlassian licenses (@atlassian.com licenses) which would be excluded in calculations in new pipeline. This means you might see a change in these numbers. | |
| As part of new pipeline improvements, a few licenses and |
|
| Accurate host/parent product insights | In the new pipeline, we rely on a more comprehensive mapping logic to derive accurate parent insights |
| In the old pipeline, we were checking if parent product of the app has a sandbox instance. In the new pipeline, we check if app is installed on sandbox site. ( Ex. A customer XYZ purchases a Jira app App1 and installs it on xyz.com. This customer has another Jira instance xyz_sandbox.com) | |
| - Old pipeline - Derive the parent product of the app, check if there is a sandbox instance associated with that parent product and flag -Yes, in this case, though the Jira app APP1 is installed on xyz.com 1 and the flag for Is sandbox is set as True | |
| New pipeline- is sandbox is set as False because xyz.com 1 is not a sandbox instance. Though the parent product Jira is installed on another sandbox instanceNote | |
| Accurate evaluation insights | In cases where there are multiple evaluations, the start date of the first evaluation and end date of the last evaluation is considered for evaluation insights. (For example: First evaluation could start on 2020-03-29 and ends on 2020-05-25.) Commercial for sometime and an evaluation starts later on 2020-07-01 and ends on 2020-07-25. |
| Opportunity size is calculated based on the end date of last evaluation (2020-07-25). |