Data Residency for Marketplace Apps Status Update

Data Residency
, , ,
1

Thank you all for reaching out with your questions about Data Residency. We recognize it’s been a while since we published any updates here in the Developer Community.

Last month, we published an update on our Developer Blog with an overview of our joint approach to evolving customer trust needs for cloud apps. This update included several upcoming changes to enhance transparency and control for cloud app customers across a number of trust topics, including data residency for apps.

Currently, Marketplace Partners can pin app data in Australia, Europe, and the USA for cloud apps built on Connect. This will allow for new installations of supporting apps to pin the app data to the same location as the host products. You can find details here.

What is the next major milestone for app data residency?

As we mentioned in the blog post and in the latest roadmap webinar, the next data residency milestone is realm migration support for Connect apps. This month, we are planning to release Connect realm migration service and APIs as an EAP for Marketplace Partners, to enable you to start integration and testing for apps to support realm migration. With these new APIs, Connect apps will be able to support migration of app data to match the parent product realm.

Please note, customers will not be able to schedule app migrations upon this upcoming EAP release. The EAP release is intended for Marketplace Partners and developers to start the integration and testing to support realm migration for your apps, prior to the customer experience release (which we are targeting for first half of 2023).

As part of this upcoming realm migration EAP release, we will update the developer documentation with updated list of supported realms, details of the new migration APIs and information on how to test app realm migration.

More details of the upcoming realm migration APIs is available in our earlier community update.

In addition to offering support for realm migration, we also plan to enable app data residency (realm pinning and migration) in Germany by the end of this year.

What is coming up on the customer-experience side for app data residency?

Following the Connect app realm migration EAP release for Marketplace Partners, we will also release features to help customers better understand and manage app data residency, including:

  1. A privacy & security tab on Marketplace listings (in the coming month we will provide more information on this feature and how to leverage it to promote your trust investments). This will help partners share app data residency information with customers while they’re searching for apps.
  2. A self-serve customer realm migration experience EAP on admin.atlassian.com to help customers view data residency options for their existing apps, and schedule their own app data migration windows.

We are actively working on both of these features right now, and expect them to arrive in the first half of 2023.

When will Forge support data residency?

Currently, Forge does not provide multi-region support. We’ve prioritized delivering data residency support for Connect apps first, to enable data residency for the largest number of Marketplace apps. However, we know that Forge support for data residency is a highly anticipated feature.

Data residency for Forge will enable multi-region support for the Storage API, providing data residency compliance to customers. As Atlassian will deliver and manage the relevant multi-region infrastructure and provisioning, it will make it easier for Forge-hosted apps to offer data residency.

We are working on plans to get data residency on Forge, and below are the factors affecting the timeline:

  1. We are currently in-progress to integrate with a new internal data store for Forge hosted storage. This new data platform will help us to deliver additional storage capabilities for Forge, including support for structured data and query-by-value (more details here). Data residency support for Forge-hosted storage will be our key focus area after we have finished the migration to this new data store.
  2. Multi-region support for Forge hosted storage may result in latency, due the geo-distributed architecture of the user, Forge’s hosted compute, and Forge’s hosted storage. To address this, we are planning to support multi-region compute for Forge to ensure low latency when Forge data residency is available. Multi-region compute is identified as a pre-requisite for Forge data residency, and we are still assessing the delivery timelines.

This is a complex body of work, and we want to be transparent about the considerations and dependencies. We will provide periodic updates as we make progress and lock-in timelines.

App data residency roadmap

As we continue to optimise the data residency solution, we will introduce additional improvements to enhance the customer experience. Some of these proposed improvements may require additional actions from app developers. Below are a few areas we plan to explore in the coming 12 months:

  1. Data storage classification for apps: We plan to introduce new parameters within the app descriptor/manifest for app developers to declare if an app stores / does not store in-scope data, and if an app stores in-scope data within the host products supporting data residency (Jira/Confluence). This will provide customer admins with a more complete view on admin.atlassian.com regarding their installed app’s data residency status.
  2. Email notifications for app migrations: Upon initial release of Connect realm migration service, app migrations status will not be included in the existing email notifications for customers. We plan to add app migration information to email notifications in future updates.
  3. Tenant pinning API: We are currently looking into feasibility of providing an API for Marketplace Partners to identify the current pinned location of a tenant’s host product.
  4. App data migration estimations: We plan to introduce a new API for apps to provide a time estimate for an app data migration for a particular tenant/installation. This will aim to align the customer experience between the host product migrations and app migrations, and ensure the customer has the relevant information to plan for any required downtime.
  5. Realm pinning persistence for app re-installs: We are looking into updating the existing realm pinning flows to enable realm persistence during an app re-install. This will ensure existing app data sets remain in the same realm as the app installation. Further, the pinned realm for an installation can only be changed by the customer using the realm migration service.
  6. Migration of host product and apps within a single window: Upon initial release of the Connect realm migration service, apps will be requested to migrate after the host product has successfully migrated, in its own migration window. However, we are looking into combining the product and app migrations within a single window to make the migration experience simpler.
  7. Minimising downtime for customers: Initially, apps will be provided a 24 hour maximum migration window to complete the data migration. In future, we will explore reducing this to minimise the amount of downtime for customers. Further, we will explore feasibility of supporting live migrations of app data to eventually eliminate downtime for customers during product and app migrations.

The scope and timelines for these are still being determined. Once we progress with these items, we will reach out to the developer community to gather feedback and keep you updated on target delivery dates.

Where can I go for updates on this topic?

We’ll discuss Forge data residency in the Forge Quarterly Roadmap Webinars, and data residency more broadly in the Marketplace Quarterly Roadmap Webinars. Further updates will also be provided in the Partner Portal (particularly in our trust hub). We will also let you know about documentation changes in the Marketplace change log.

10 Likes
2

Hey @SushantBista ,

thanks for the heads-up! Two questions that come to my mind:

  1. Will support for data residency and realm support become a requirement for the Fortified badge?
  2. Will Atlassian roll out the new privacy tab in the Atlassian marketplace (announced at Marketplace Roadmap Webinar) before Forge supports data residency? I fear that Forge apps will face a situation where we can’t comply with the highest standards in that new overview because Atlassian has to provide the data residency support for the platform.

Cheers
Julian

1 Like
3

Hey @JulianWolf, thanks for your questions.

  1. Currently data residency is not part of the Cloud Fortified program requirements. The Cloud Fortified team evaluates the customers’ needs and expectations from the program continuously. One change that is coming to Cloud Fortified program in the near future is filling out Privacy & Security tab information, which will replace the Security Self Assessment requirement. If there is a plan to make data residency support a program requirement in the future, Cloud Fortified team will collect input from partners and share the plans in advance to make sure that partners will have enough time to meet the new requirements.

  2. Customers and Marketplace Partners have an urgent need for a centralized place to find and share key trust information, and we want to meet this need as soon as possible to relieve some of the customer pain we’re observing. Unfortunately, that does mean the Privacy & Security tab will go live to customers before Forge storage API supports data residency.
    Forge apps that currently store data exclusively within Jira and Confluence (e.g. entity properties) are already data residency compliant, and this will be reflected in the tab. The tab will also highlight other benefits of building on Forge to customers, like using hosted storage. Forge will ultimately help partners offer many of the trust features highlighted on the tab more easily, although our progress in this area will come in stages.
    We acknowledge that this situation is not ideal and we’re exploring ways to fast-track data residency support for Forge storage API availability, and will let you know when we have more clarity on the Forge data residency timelines.

Thanks,
Sushant

4

Thank you @SushantBista for your answer.

This is more than a bummer. For two years Atlassian advertises Forge as the new platform being “ready for enterprise scale”. Already today vendors following the guideline to not start new Connect apps face huge disadvantages being in competition with existing apps on the marketplace as many features are still missing.

I understand that Atlassian now is up to flag apps with missing data residency in the privacy tab which will bring Forge vendors in an even more miserable situation.

We know that Data Residency is an important requirement for customers. We get many questions around that topic. To this point we were happy to tell the customers that we do our very best to comply with the highest app standards Atlassian offers by using Forge.

I feel like Atlassian is enforcing a situation where their new framework can’t follow their own rules introduced by the new Privacy tab. I’m clueless and understand this information as a hint that we should evaluate shifting ongoing app developments back to Connect.

My suggestion would be that Atlassian rethink their priorities. The outlined roadmap will bring Forge in an unfavourable position compared to Connect, which contradicts what Atlassian told partners the last years. I’m unsure if that is the intention of this announcement but this will definitely throw back the adoption of the Forge platform as a whole.

9 Likes
5

Thank you @JulianWolf for your feedback.

Firstly, I want to emphasise that Forge is still our number one priority and we are continuing our efforts to make Forge our developers’ preferred choice, by providing more of what developers need through the platform.

We decided to decouple the release of the new Privacy & Trust tab from Forge data residency, as we were not able to align the release plans / timelines for the two projects. While not ideal, this decision was to ensure we ship immediate customer value as mentioned in my earlier comment.

We acknowledge the concerns and sentiment you have raised and we certainly appreciate the position you are in. If data residency is unavailable due to Atlassian’s roadmap, we will be sure to call this out in the Privacy and Security tab. We are exploring ways to address this, while continuing to meet immediate customer needs. We will keep you updated as we make progress.

Thanks,
Sushant

1 Like
6

Thank you for elaborating on this @SushantBista – I appreciate it.

We’ll rely on that and assume that Atlassian won’t flag vendors and apps that are limited by Forge in terms of Data Residency support.

Cheers
Julian

3 Likes
7

Hi @SushantBista, I have a question about data residency.

I understand about the “storing” part, meaning the database of my app should be in the region/country that the customer resides.

What about the “transfer” part? Is there any requirement for that?

For example, if we are to support data residency in our app for EU customers: our database will have to be in EU. Do we have to serve the data from EU (from our backend server) as well? If the “transfer” part is required, I understand that the backend server will have to be in EU.

Can you explain this one for me (or can guide me to any document)?

1 Like
8

Hi @TrungLe,

Generally, it will be up to the Marketplace vendor to define what is in-scope for an app’s data residency policy. Details about data storage and processing should be documented and made available to customers.

For reference, data in transit (up to 30 days), including data being processed and not at rest is currently not in-scope for Atlassian products data residency. More details are available here.

Thanks,
Sushant

1 Like
9

Hi @SushantBista
have a question around Dataresidency

  1. Lets say we have a Connect App( NOT complaiant with Data residency) and we have a customer using it (has some data of the customer stored with APP).
  2. Later we enable Data residency for the APP and release the APP
  3. Lets say the Tenant Host is in EU region, and since APP enabled EU region we start getting traffic to
    EU based regional baseURLs (i.e to store and process the Data in EU) one question here is how do we identify Proximity vs User pinning a region
  4. How do we handle the data created when the app was not Data residency complaint ? do we handle migration of it too (i.e copy the data created in step 1 global to EU region ?)
    or is it guarenteed to be treated as global as mentioned here How to migrate customers to the correct region after releasing data residency support - #4 by SushantBista

please let us know(questions asked in 3&4 above) this seems to be critical information.

Thanks,
Purna.

10

Hi @PurnaChandraBoyapati!

To answer your questions. It is true that traffic for an unpinned tenant is based off the proximity of the product (i.e. if it’s located in the EU, then Connect will route installations to your EU-declared app server in the descriptor). And if pinned, to the relevant pinned declared app server.

Now, unfortunately it is not possible to distinguish between installation routed based off proximity versus an installation routed based off the product being pinned. However, there has been talks about providing a new product API accessible by apps which enable app developers to obtain the residency of the tenant (whether it is pinned or not - and if pinned, to which location).

To answer #4, this is the intention of Data Residency migrations introduced in this post. It provides a mechanism for customers to trigger a migration for their apps, and subsequently a mechanism for app developers to listen to the migration lifecycle to move data between realms.

Now, an installation will continue to stay in the same realm unless an uninstallation happens. Then the routing rules will apply again. Subsequent upgrades and so forth will not redirect traffic to a new realm - even though it is compliant. This is explained here.

Hopefully that answers your questions! Please don’t be afraid to reach out if you have any further questions!

3 Likes
11

Thanks for the clarification, I got the concept of realm being fixed once the user pins the Instance to a region but I have a minor variation (i.e before enabling DaRe and after enabling DaRe)

  • lets say a Customer is using the app before data-residency is introduced by Atlassian or the App Provider
  • on subsequent updates after “Dataresidency enabled by APP”
    – can I assume the App on that Instance is in Global realm ?
    – or is there a chance that it still has to be migrated implementing the latest hooks ? considering the proximity criteria it might fall into a specific Realm(US EU AU…).
12

Hiya @PurnaChandraBoyapati!

So to elaborate on that scenario to ensure we’re on the same page.

We’re talking about installations that occurred prior to Data Residency being introduced by Atlassian or being introduced by the app.

In this case - yes, all installations would have gone through your declared baseUrl and would be considered a global realm installation. These installations continue to go through the global realm so is a fair assumption but the same rules apply - if an uninstallation and installation happens and the app then supported a realm desired by the product, then it is possible for the proximity criteria or pinning rules to kick in.

WRT to the migration question. I’m not quite sure I understand - irrespective of where the app is located. The intention is still for customers to move apps which are not on the same realm as the product to the said-desired realm. So if an installation is in the global, it will show up as so, and consequently, customers may request an app migration if your app supports the desired realm and the migration hooks.

Proximity may already put this into the desired region for the customer - in which case, the app may not need to move at all. However, it might also be that proximity may be “EU”, but the customer desires “US” and then pins the product to the “US” - in which case, a app migration may be required by the customer.

I hope that answers your questions!

1 Like
13

yes, clear to me.

thank you.

14

@JackyLo
can we release an App with below conditions?

  • with Dataresidency support
  • but NO realm-to-realm migration support ?
1 Like
17

Hi @PurnaChandraBoyapati,

To fully support the various use cases of data residency both the regionBaseUrls and the dare-migration field have to be added to the Connect descriptor.

This is how they will be used:

  • the URLs defined as regionBaseUrls control the location/realm when an app is installed. If a customer moved a product to a specific location, when an app is installed, the corresponding location from the Connect descriptor will be selected. Even if the product has not been been moved to a specific location, the regionBaseUrls is used at app installation time if the app supports the location where the product is provisioned. More information can be found on the (Data residency developer documentation).
  • the lifecycle dare-migration field are used when a customer requests a migration for the app data. This happens when the app was installed in the Global location because either the product was not migrated yet or because the app didn’t support data residency. In cases like these, the customers will have to request the app data to be moved. The Advance Notice: Connect Data Residency Customer Rollout topic shows the UX for customers and the data residency migrations contains the details of the required hooks.

To recap, while there is no validation on the descriptor and only providing one or the other field is technically possible, both are required to support all possible scenario.

Hope this helps,
Caterina

3 Likes
18

Hi @ccurti , I hope you’re well.

I’m wondering if we could have an update about the roadmap shared above? I’m particularly interested in this item: Realm pinning persistence for app re-installs. That’s currently a blocker for us (Zephyr Scale) to release data residency.

Thank you,
Vitor

19

Hey @VitorPelizza,

Apologies for the delayed response, this slipped by. We recently announced that Data Residency was available for Admins under a beta label. Subsequent to this release, we have been working on ways better communicate the status of Apps, particularly where they do not store data outside of Atlassian.

While we certainly want to explore the persistence of realms on app reinstallation, it is not something which we have planned for the short to medium term. It would be great to learn more about why you see this as a blocker and if there are any other considerations we should make as we explore this functionality. Please feel free to book some time in my calendar to connect and discuss this (or general data residency in Connect) further.

Thanks,
Sean

20

Hi Sean,

Thank you for your answer. Unfortunately, this is not the answer I was hoping for. Including on this page but also seen in other materials, including during conversations at AppWeek@Berlin, this item is on the roadmap. I understand it’s never been on the short-term roadmap, but this conversation have been going on for 6 months or so now, so I was hoping that this would be somewhere near to get prioritised.

This behaviour is critical for us to release data residency for Zephyr Scale. Without this bug fix (it is a bug in my opinion), our customers might get pointed to an empty installation of Zephyr Scale if they reinstall their app after we release data residency. If that scenario happens, we won’t be able to move their data over, particularly not without intervention.

Unfortunately, I can’t find a suitable time in your agenda as I’m in the UK. Would you mind to share some European-friendly times?

Many thanks,
Vitor Pelizza

2 Likes
21

What do we do with dare-migration if all app data is only stored in Atlassian storage that already supports data-residency and location migration?

22

Hi @daviddrawio,

we shared this in the Advance Notice: Connect Data Residency Customer Rollout topic

  • What if I store data in Jira and Confluence?
    • If you store data exclusively within Jira and Confluence, the data will inherit data residency controls already available for these products, therefore you do not need to support Connect Data Residency realm pinning and migrations.

However, as a few commented there, with the current UI the app will appear as Not Eligible to be pinned or migrated. The following is presented to the users when browsing to that tab and the data stored exclusively within the Atlassian product is the first item in the bullet list:

image
image1902×824 89.8 KB

This is something that the team is considering improving (as mentioned here).

Caterina

2 Likes