Advance Notice: Connect Data Residency Customer Rollout

Context

In November 2022, we announced the early access program for Connect app realm migration. During this period, partners have had access to the APIs and underlying service in order to start building these capabilities into their apps. We are now approaching the release of the corresponding customer UI in the admin panel, which will begin a slow rollout on March 29, 2023.

Expected Timeline

On March 29 2023, the ability to use realm pinning and realm migration functionality will surface in admin.atlassian.com and we anticipate ~1,000 customers will see their existing apps’ data residency options. This functionality will be tagged with a ‘beta’ label. There will be no formal announcement drawing attention to this functionality throughout the duration of the Early Access Program.

Realm Migration for Connect apps will remain in the Early Access Program stage as it is today, and we plan to officially transition into the Preview release stage once we have tested the demand and functionality with early adopter partners and customers. Although Realm Migration will still be considered in EAP, partners who have successfully tested Realm Migration and have confirmed it is functioning as expected have the option to make it available to customers. This way, customers will have the chance to test this out as part of their Beta as well.

We are targeting June 2023 for the partner Preview* release and full rollout of the Customer UI Beta.

Anticipated customer views per month Target date/month
~1,000 customers

~2,000 customers

~4,500 customers
March 29, 2023

April 30, 2023

June 30, 2023

*Preview means that the feature is available for production use, and that a 1-3 month deprecation period and notice is provided for any breaking changes that may be introduced.

Customer Experience

The customer facing UI will show details regarding realm pinning and realm migration. If an installed app supports realm migration, it will show which realms are available for pinning as well the current pinning status, including whether the app is compliant based on the host product pin location. If the app does not support realm migration, then it will show a status saying ‘Not Eligible’.

With this first release, we anticipate ~1,000 customers with self-service data residency enabled for JSW, JSM, or Confluence Cloud will see their existing apps’ data residency options on admin.atlassian.com.

These options include:

  • Pinned: Apps have data pinned to a realm.
  • Eligible: Apps are eligible to move to the same realm as the parent product.
  • Not Eligible: Apps either do not support data residency in the same region as the parent product, or do not need to independently support data residency. Check the app listing/app privacy policy for more information about these apps.

If the customer’s parent product (Jira/Confluence) has data pinned to a region, customers will also see a new option to “Move apps” to the parent product region from within the data residency tab.

Clicking “Move apps” will allow customers to set a window of time to migrate all “eligible” apps that:

  • support data residency in the region of the host product and
  • support automated realm migration

For example, if a customer has their Jira Cloud data pinned to the United States, they can click “Move apps” to migrate data for apps that offer data residency in the United States.


Click into the product to see attached apps


Customers can tab between eligible apps, pinned apps, and ‘not eligible apps’ or apps that do not support (or do not need) data residency


Customers can schedule a window of time to migrate app data for apps that support realm migration

All apps that can move will do so during a pre-set window of the customer’s choosing. Apps will only be able to migrate data while the host product is offline, so customers should plan for some downtime during this process.

Feedback & questions

As we begin to see customers leverage your Connect app’s current realm pinning/migration capabilities, we’ll be monitoring both partner and customer feedback closely. Please leave any questions or feedback on this post directly, or send a direct message to me here on the developer community.

FAQ

  • Why are we starting to release this now?

    • Realm pinning has been available for some time now and even though partners have made substantial investment in it, customers don’t currently have a way to see which of their apps are pinned. With the new UI, partners who have invested in realm pinning can begin to highlight this work to customers. We believe that starting to observe customer adoption through a slow rollout is the best next step to help allow partners to reap the benefits of their investments.
  • What if I store data in Jira and Confluence?

    • If you store data exclusively within Jira and Confluence, the data will inherit data residency controls already available for these products, therefore you do not need to support Connect Data Residency realm pinning and migrations.
  • When can I indicate support for a realm?

    • If you indicate support for a realm, it means your app stores the relevant, in-scope data only in that realm - any in-scope data that your app stores must be stored in that realm. You should not declare the regionBaseURL for any realm unless you use storage infrastructure in that region to store in-scope customer data.
  • What if I don’t store any customer data?

    • It is recommended to support data residency if your app stores end user data. Atlassian has outlined what is considered in-scope for data residency for its products here. If your app does not store any data, then your responses to your Privacy and Security should reflect what data your app uses/stores to help inform customers.
  • What should customers do if they see that an app is ‘not eligible’ to move and are not able to use the Realm Migration functionality?

    • If a customer sees that an app is not eligible to pin data to the host product realm, we will advise that they check the app listing for more information. This is because the Privacy Policy (and eventually the Privacy & Security tab) expands on the various ways apps handle data.
  • How will customers know this is an EAP?

    • There will be a ‘Beta’ label for Connect app data residency in the customer UI, indicating that this functionality is not yet fully in production.
  • Is there an update on Forge data residency?

3 Likes

Hello,

Should we expect any updates before customer rollout in regards to concerns for re-pinning via uninstall raised in the previous topic? We’re seeing it as a big risk for data consistency

Hi,

I’d be interested, how a sort of “fake” migration could be technically accomplished, i.e. will there be provided some way to posteriorly make an app pinned, provided that we already store the respective data in the desired region?
This means, that an actual migration shall not take place, in this particular case it’s only an administrative task. Should we still fake a real migration, by implementing the entire (regular) workflow (via lifecycle hooks), or will be there a simpler way to do this and register/show the app as pinned for the customer at hand?

Thanks,
Marton

1 Like

In this case do you still need to declare regionBaseURLs in the descriptor?

For apps that store data only in Confluence/Jira I’d want to indicate by default that they’re already migrated to a data region in the admin dashboard. Seems as if these apps instead will be grouped in the “not eligible” category? If so that’s some poor signalling to customers.

5 Likes

Hey @lexek-92,

Appreciate your concern regarding realm persistence for re-installations (i.e. re-pinning on reinstall) and understand the implications it can have when a previous installation uninstalls/reinstalls an app. As @SushantBista has previously shared, this is something which we are exploring and are seeking to understand better.

If any active participants of data residency have been experiencing challenges due to this, I would be open (and keen) to discussing and exploring the impacts which it has had. @lexek-92 I’ll reach out to discuss further.

1 Like

Hey @marton.kelemen,

Thanks for the question, just want to clarify my understanding correctly. In this circumstance you are already hosting the data in a specific region which is not one the same one which Connect is aware of (potentially before enabling data residency)? In this circumstance, you’d want to ensure that Connect is aware of which region that data is already hosted in, so that it could be represented as pinned and eliminate the need for a customer to have to request a migration?

With regards to the lifecycle events, the initial /schedule would inform you of the target region. Once the migration starts and customer data has been migrated (or in this circumstance, exists) within that region, you would respond to the /status request with ready-to-commit.

Where do vendors have to indicate that an app only stores data in the host product (region).

  1. Security questionnaire (is this the only place so far?)
  2. Connect app descriptor?
  3. ??
1 Like

Hi @UlrichKuhnhardtIzym1,

I assume you look for this: Privacy & Security tab
Here you can provide the necessary information:
See also the respective API method: if I’m correct, the appTrustInformation -> dataAccessAndStorage -> isSameDataProcessedAndStored is the variable that tells about this.

2 Likes

Hey @SeanBourke,

Thank you for your prompt reply.
I can confirm, your understanding is correct.

I can also provide you with a very typical use case:

Our app registered the customer to the desired location (e.g. to the EU) based on Atlassian’s routing choice (i.e. based on geolocation), even though the instance was initially not pinned. Later on, if the instance gets pinned by the customer, we still have to do an official migration process for our app’s data, instead of just labeling the app’s data as already migrated.

So, this means that our app has to go through the standard migration flow, even if there is no real need for moving any data? Of course, we can shortcut at least the heavy part, as you wrote, but…
I still have concerns regarding this.

First of all, what is not clear to me is that how much system downtime can such a “fake” migration induce. If I’m right, the host product must be shut down/“freezed” during the data migration process. Are there any other potential side-effects, too? (provided that otherwise our app would be the only one participating this data migration session, these effort are all in vain)

There is another, not that much of a technical issue, but still an important aspect: how can customers get informed about the opportunity/necessity of a data migration at all? I think, following this concept they have to trigger all the events and then see the outcome, therefore, if they do not take the initiative, they won’t be aware of the fact, where their data are stored. Of course, if it’s not crucial to them, then we also can live with it, but a bit more proactive attitude could help a lot…

Apart from these, I could imagine that by adding an extra data storage location field to the migration overview (for which, the response could be obtained by another hook preliminary to the migration => similar to the /schedule hook), the admins could assess their apps’ statuses better, without the necessity to initiate an entire migration in order to get the answers for such questions.
As an alternative, an app could indicate in the response given to the /schedule event that the data are already migrated to the specified location, so subsequent hooks shall not be called. However, this could only tell the user, whether their data are already in a requested/target location, so the first approach seems to be more generic and user-friendly for me.

Anyway, if the implementation of such an assessment step would require too much effort, then we’ll go for the suggested solution. It can surely work, even if it’s suboptimal in some scenarios.

Best,
Marton

1 Like

Hey @marton.kelemen,

Thank you for the well thought out reply. My apologies for the delayed response, I wanted to ensure that we address each of the questions you’ve asked - hopefully the below helps.

If your app supports a regionBaseUrl of EU and Connect directs traffics for a customer’s app installation to EU based upon their proximity to that realm, then they will be deemed as being pinned to the EU realm.

  • If the customer’s product exists in (or later moved to EU), then that customer would see the app as Pinned to the EU region.
  • If the customer moves their product to a different realm which you also support in your Connect descriptor, then they will see your app as Eligible to move with a current location of EU
  • If the customer moves their product to a different realm which you do not support in your Connect descriptor, then they will see your app as Not Eligible to move with a current location of EU
  • If the customer had installed the app prior to any data residency realms being supported, then they would likely be directed to your baseURL and will appear as Eligible given they will be deemed as being in Global (or Not Set).

The /status endpoint is queried on a 15 minute interval, meaning that in this circumstance, the maximum duration of product downtime would be 15 minutes. It’s worth noting that the scheduled migration period can include multiple apps and the product downtime is impacted by the longest running migration task; so if a different app takes longer to migrate in the window, then the product will be unavailable until that app also completes its migration. I’d be keen to understand if you feel that for live migrations (where you have to move data between regions), if a 15 minute polling period is sufficiently frequent?

At the moment, this is provided through the via the Admin UI to provide admins visibility of where app data is currently located. A previously proposed improvement to have prior awareness (i.e. inverse app awareness) of a mismatch with a customer’s realm based upon the feasibility of a tenant pinning API would also provide the capability to support customers in understanding when an opportunity exists to realign data. There is no available scope or timeline available for this at the moment, however.

Appreciate this feedback and understand how it could create efficiencies in the process. It would be great to understand the frequency you have to perform fake migrations once this is live, as this would inform the potential for an improved customer experience. As noted earlier, the minimum downtime for a customer’s product is 15 minutes (pending all app migrations are completed in that window).

1 Like

^ can I please get an answer on this one?

Hey @nathanwaters,

If you only store data in confluence and Jira, then you would not need to declare the regionBaseURLs in the descriptor.

Appreciate your concern regarding the signalling of apps in the Not Eligible category and have shared this feedback with the Admin UI team. Customers will be guided to review app’s Privacy and Security tabs to better inform them about what data an app utilises and how it stores data.

2 Likes

Thanks. I think apps that only store data in Confluence/Jira should be under their own category.

If it’s just a link to the security tab on the marketplace, customers are 100% not going to bother going through those steps… they’ll just incorrectly assume the app is not handling regional data.

So you’re punishing apps who have gone to extreme efforts to keep all data inside Atlassian’s stupid 32kb content/entity property limits.

If you’re going to place such apps under the “Not Eligible” category then it makes perfect sense for me (and every other vendor) to add fake regionBaseUrls matching the baseUrl so the reality of how data is handled gets properly conveyed to customers.

Many of the largest apps on the marketplace already do this: regionBaseUrls are identical to the baseUrl in their descriptor. Very easy to find examples: start with the apps in the screenshots at the top of this thread.

Is this the suggested approach to correctly signal so my apps don’t get thrown into the “Not Eligible” category?

4 Likes

I totally agree with Nathan. Please make another category for apps that store data in Confluence and Jira. In fact, that category should be prioritized.

3 Likes

Also agree with @nathanwaters ,
Apps that store data in Jira/Confluence can offer data residency, in contrast to Forge apps. That should be highlighted for customers.

2 Likes

@nathanwaters , @chhantyal and @marc thank you all for your feedback. I’ve shared this with the customer facing UI team and we are exploring options to better represent an app’s status within the Admin UI. While this will not be available within the initial beta release to customers, we will provide more details as this progresses. In the interim, I recommend ensuring that you have accurately completed your Privacy and Security tab responses, as customers will be guided to review these to better understand how apps manage their data.

1 Like

Hello from Stiltsoft Europe!

We also would like to contribute to the subject.

As one of our add-ons Table Filter and Charts for Confluence doesn’t access, store, or otherwise process any personal data since all the page content is stored within the Confluence page, the option for it is to be Not Eligible.

However, our customers are deeply concerned about data transition, which in fact takes place through our servers on occasions of content exporting and for some of the macros parsing. These cases are directly related to customer data handling and as such are part of the Data Residency.

In light of that, we believe it would be better actually not to consider such apps as ours Not Eligible but instead add some kind of indication/flags in the app descriptor pointing at Data Residency actually supported, with the data stored only in Confluence.

Hopefully, this matter will be accounted for accordingly

Many thanks!

5 Likes

:wave: @NikitaKamai and team,

I want to clarify that there are cases where the app will be presented as Pinned:

  • a customer site is located in US based on their origin of access when the site was created → the customer installed the app after the US location was supported in the descriptor via the regionBaseUrls tag → the customer then moved (or moves) the product to the US location
  • a product has been moved to the US location → the customer installed the app after the US location was supported in the descriptor via the regionBaseUrls tag

The same applies to the other supported regions by the app.
This is because the app is currently supporting data residency pinning based on the descriptor via the regionBaseUrls tag.

If, instead, a customer moves the product to a different location from the one where the site was created, the app will only be shown as Eligible once it supports data residency migrations.

In the case of this app, once data residency migration is supported, the Not Eligible option will only be shown for those regions that are not supported in the descriptors.

If your plan is to support all regions and the migration option, Not Eligible will not be one of the possible options for this app.

As Sean shared above, we are looking into options for a better representation of the app’s status. In the meantime, please provide the answers to the Privacy & Security questions for customers to review the data residency offering of the app.

Hope this helps,
Caterina

1 Like

The gist here is that our app does not store any customer data, as it’s actually stored by Confluence as part of its page, for example. Therefore, Table Filter and Charts add-on requires no data residency migration in fact. While so far it looks as now the status for the app is to be Not Eligible because of this, which in reality has nothing to do with how things are.

We will greatly appreciate if your team could consider this circumstance accordingly.

Thank you kindly,
Nikita

1 Like

Hi @NikitaKamai,

I appreciate that similar feedback has been shared by the community and this has been shared with the customer facing UI team. We are currently exploring how we can better represent app statuses to customers. In the interim, I recommend ensuring that you have accurately completed your Privacy and Security tab responses, as this is likely to be used by customers to better understand how an app utilises and stores their data.

While adoption of data residency and realm migrations has primarily been focused around in-scope data, I also appreciate that some partners have adopted data residency within their Connect apps for other reasons, such as server proximity to customers. If you already support data residency, or believe it would be valuable, extending that to also support realm migrations helps to ensure that your app pinning is aligned to the customer’s product.

1 Like