Data Residency for Marketplace Apps Status Update

Thank you all for reaching out with your questions about Data Residency. We recognize it’s been a while since we published any updates here in the Developer Community.

Last month, we published an update on our Developer Blog with an overview of our joint approach to evolving customer trust needs for cloud apps. This update included several upcoming changes to enhance transparency and control for cloud app customers across a number of trust topics, including data residency for apps.

Currently, Marketplace Partners can pin app data in Australia, Europe, and the USA for cloud apps built on Connect. This will allow for new installations of supporting apps to pin the app data to the same location as the host products. You can find details here.

What is the next major milestone for app data residency?

As we mentioned in the blog post and in the latest roadmap webinar, the next data residency milestone is realm migration support for Connect apps. This month, we are planning to release Connect realm migration service and APIs as an EAP for Marketplace Partners, to enable you to start integration and testing for apps to support realm migration. With these new APIs, Connect apps will be able to support migration of app data to match the parent product realm.

Please note, customers will not be able to schedule app migrations upon this upcoming EAP release. The EAP release is intended for Marketplace Partners and developers to start the integration and testing to support realm migration for your apps, prior to the customer experience release (which we are targeting for first half of 2023).

As part of this upcoming realm migration EAP release, we will update the developer documentation with updated list of supported realms, details of the new migration APIs and information on how to test app realm migration.

More details of the upcoming realm migration APIs is available in our earlier community update.

In addition to offering support for realm migration, we also plan to enable app data residency (realm pinning and migration) in Germany by the end of this year.

What is coming up on the customer-experience side for app data residency?

Following the Connect app realm migration EAP release for Marketplace Partners, we will also release features to help customers better understand and manage app data residency, including:

  1. A privacy & security tab on Marketplace listings (in the coming month we will provide more information on this feature and how to leverage it to promote your trust investments). This will help partners share app data residency information with customers while they’re searching for apps.
  2. A self-serve customer realm migration experience EAP on admin.atlassian.com to help customers view data residency options for their existing apps, and schedule their own app data migration windows.

We are actively working on both of these features right now, and expect them to arrive in the first half of 2023.

When will Forge support data residency?

Currently, Forge does not provide multi-region support. We’ve prioritized delivering data residency support for Connect apps first, to enable data residency for the largest number of Marketplace apps. However, we know that Forge support for data residency is a highly anticipated feature.

Data residency for Forge will enable multi-region support for the Storage API, providing data residency compliance to customers. As Atlassian will deliver and manage the relevant multi-region infrastructure and provisioning, it will make it easier for Forge-hosted apps to offer data residency.

We are working on plans to get data residency on Forge, and below are the factors affecting the timeline:

  1. We are currently in-progress to integrate with a new internal data store for Forge hosted storage. This new data platform will help us to deliver additional storage capabilities for Forge, including support for structured data and query-by-value (more details here). Data residency support for Forge-hosted storage will be our key focus area after we have finished the migration to this new data store.
  2. Multi-region support for Forge hosted storage may result in latency, due the geo-distributed architecture of the user, Forge’s hosted compute, and Forge’s hosted storage. To address this, we are planning to support multi-region compute for Forge to ensure low latency when Forge data residency is available. Multi-region compute is identified as a pre-requisite for Forge data residency, and we are still assessing the delivery timelines.

This is a complex body of work, and we want to be transparent about the considerations and dependencies. We will provide periodic updates as we make progress and lock-in timelines.

App data residency roadmap

As we continue to optimise the data residency solution, we will introduce additional improvements to enhance the customer experience. Some of these proposed improvements may require additional actions from app developers. Below are a few areas we plan to explore in the coming 12 months:

  1. Data storage classification for apps: We plan to introduce new parameters within the app descriptor/manifest for app developers to declare if an app stores / does not store in-scope data, and if an app stores in-scope data within the host products supporting data residency (Jira/Confluence). This will provide customer admins with a more complete view on admin.atlassian.com regarding their installed app’s data residency status.
  2. Email notifications for app migrations: Upon initial release of Connect realm migration service, app migrations status will not be included in the existing email notifications for customers. We plan to add app migration information to email notifications in future updates.
  3. Tenant pinning API: We are currently looking into feasibility of providing an API for Marketplace Partners to identify the current pinned location of a tenant’s host product.
  4. App data migration estimations: We plan to introduce a new API for apps to provide a time estimate for an app data migration for a particular tenant/installation. This will aim to align the customer experience between the host product migrations and app migrations, and ensure the customer has the relevant information to plan for any required downtime.
  5. Realm pinning persistence for app re-installs: We are looking into updating the existing realm pinning flows to enable realm persistence during an app re-install. This will ensure existing app data sets remain in the same realm as the app installation. Further, the pinned realm for an installation can only be changed by the customer using the realm migration service.
  6. Migration of host product and apps within a single window: Upon initial release of the Connect realm migration service, apps will be requested to migrate after the host product has successfully migrated, in its own migration window. However, we are looking into combining the product and app migrations within a single window to make the migration experience simpler.
  7. Minimising downtime for customers: Initially, apps will be provided a 24 hour maximum migration window to complete the data migration. In future, we will explore reducing this to minimise the amount of downtime for customers. Further, we will explore feasibility of supporting live migrations of app data to eventually eliminate downtime for customers during product and app migrations.

The scope and timelines for these are still being determined. Once we progress with these items, we will reach out to the developer community to gather feedback and keep you updated on target delivery dates.

Where can I go for updates on this topic?

We’ll discuss Forge data residency in the Forge Quarterly Roadmap Webinars, and data residency more broadly in the Marketplace Quarterly Roadmap Webinars. Further updates will also be provided in the Partner Portal (particularly in our trust hub). We will also let you know about documentation changes in the Marketplace change log.

6 Likes

Hey @SushantBista ,

thanks for the heads-up! Two questions that come to my mind:

  1. Will support for data residency and realm support become a requirement for the Fortified badge?
  2. Will Atlassian roll out the new privacy tab in the Atlassian marketplace (announced at Marketplace Roadmap Webinar) before Forge supports data residency? I fear that Forge apps will face a situation where we can’t comply with the highest standards in that new overview because Atlassian has to provide the data residency support for the platform.

Cheers
Julian

1 Like

Hey @JulianWolf, thanks for your questions.

  1. Currently data residency is not part of the Cloud Fortified program requirements. The Cloud Fortified team evaluates the customers’ needs and expectations from the program continuously. One change that is coming to Cloud Fortified program in the near future is filling out Privacy & Security tab information, which will replace the Security Self Assessment requirement. If there is a plan to make data residency support a program requirement in the future, Cloud Fortified team will collect input from partners and share the plans in advance to make sure that partners will have enough time to meet the new requirements.

  2. Customers and Marketplace Partners have an urgent need for a centralized place to find and share key trust information, and we want to meet this need as soon as possible to relieve some of the customer pain we’re observing. Unfortunately, that does mean the Privacy & Security tab will go live to customers before Forge storage API supports data residency.
    Forge apps that currently store data exclusively within Jira and Confluence (e.g. entity properties) are already data residency compliant, and this will be reflected in the tab. The tab will also highlight other benefits of building on Forge to customers, like using hosted storage. Forge will ultimately help partners offer many of the trust features highlighted on the tab more easily, although our progress in this area will come in stages.
    We acknowledge that this situation is not ideal and we’re exploring ways to fast-track data residency support for Forge storage API availability, and will let you know when we have more clarity on the Forge data residency timelines.

Thanks,
Sushant

Thank you @SushantBista for your answer.

This is more than a bummer. For two years Atlassian advertises Forge as the new platform being “ready for enterprise scale”. Already today vendors following the guideline to not start new Connect apps face huge disadvantages being in competition with existing apps on the marketplace as many features are still missing.

I understand that Atlassian now is up to flag apps with missing data residency in the privacy tab which will bring Forge vendors in an even more miserable situation.

We know that Data Residency is an important requirement for customers. We get many questions around that topic. To this point we were happy to tell the customers that we do our very best to comply with the highest app standards Atlassian offers by using Forge.

I feel like Atlassian is enforcing a situation where their new framework can’t follow their own rules introduced by the new Privacy tab. I’m clueless and understand this information as a hint that we should evaluate shifting ongoing app developments back to Connect.

My suggestion would be that Atlassian rethink their priorities. The outlined roadmap will bring Forge in an unfavourable position compared to Connect, which contradicts what Atlassian told partners the last years. I’m unsure if that is the intention of this announcement but this will definitely throw back the adoption of the Forge platform as a whole.

8 Likes

Thank you @JulianWolf for your feedback.

Firstly, I want to emphasise that Forge is still our number one priority and we are continuing our efforts to make Forge our developers’ preferred choice, by providing more of what developers need through the platform.

We decided to decouple the release of the new Privacy & Trust tab from Forge data residency, as we were not able to align the release plans / timelines for the two projects. While not ideal, this decision was to ensure we ship immediate customer value as mentioned in my earlier comment.

We acknowledge the concerns and sentiment you have raised and we certainly appreciate the position you are in. If data residency is unavailable due to Atlassian’s roadmap, we will be sure to call this out in the Privacy and Security tab. We are exploring ways to address this, while continuing to meet immediate customer needs. We will keep you updated as we make progress.

Thanks,
Sushant

1 Like

Thank you for elaborating on this @SushantBista – I appreciate it.

We’ll rely on that and assume that Atlassian won’t flag vendors and apps that are limited by Forge in terms of Data Residency support.

Cheers
Julian

3 Likes

Hi @SushantBista, I have a question about data residency.

I understand about the “storing” part, meaning the database of my app should be in the region/country that the customer resides.

What about the “transfer” part? Is there any requirement for that?

For example, if we are to support data residency in our app for EU customers: our database will have to be in EU. Do we have to serve the data from EU (from our backend server) as well? If the “transfer” part is required, I understand that the backend server will have to be in EU.

Can you explain this one for me (or can guide me to any document)?

1 Like

Hi @TrungLe,

Generally, it will be up to the Marketplace vendor to define what is in-scope for an app’s data residency policy. Details about data storage and processing should be documented and made available to customers.

For reference, data in transit (up to 30 days), including data being processed and not at rest is currently not in-scope for Atlassian products data residency. More details are available here.

Thanks,
Sushant

1 Like