Data residency realm migration for Connect apps: Early Access Program

Announcements
, ,
1

Today, we are happy to announce the launch of the Early Access Program (EAP) of data residency realm migration for Connect apps. This EAP release includes a new realm migration service and associated APIs, which will enable you to start integration and testing to support migration of app data to match the parent product realm. For more details of this EAP, please refer to developer documentation.

If you have any questions or would like to provide feedback, please reach out to us by leaving a comment on this post.

If you face technical issues with APIs or bugs, please raise via our developer support desk.

Read the full blog announcement here.

Update on supported regions of Realm Pinning

In addition, we have added Germany as one of the supported realms for Connect Data Residency. To see our list of supported regions, please read the documentation.

7 Likes
2

Hi, thanks for the heads-up!

I just looked at the docs and didn’t see anything related to the standard lifecycle events. Not sure why (maybe I got this from some older announcement etc.) but I thought that the migration target realm app instance would receive the standard install and enabled lifecycle webhooks (and the source realm app instance would receive the disabled and uninstall events).

If not I suppose the tenant record migration needs to be done during the scheduled migration phase.

Could you please confirm either case?

This is for an app that only stores tenant records in our storage.

Thanks,
Jens

2 Likes
3

@SushantBista thank you for the announcement.

Is there documentation available which data are considered in scope for data residency for apps? I’m asking especially about:

  • Are the install data and shared secret data which are in scope?
  • Are data like the accountId in scope, given that Atlassian stores them in the US irrespective of data residency settings?
  • Does data residency concern only storage, or also processing? As an example, if an app stores all data in Confluence, and processes e.g. page data from Confluence by reading the data and writing the data back to Confluence: Is this processing to happen in the data residency realm?
2 Likes
4

Hi @jens!

So just to elaborate a bit on the existing data residency behaviour for Connect.

Currently, after an instance is pinned, is it entirely possible for an app to move realms. This behaviour is facilitated by an uninstallation by the customer, followed by a reinstallation - thus causing the old realm to receive a disabled and uninstall lifecycle event, and the new realm to receive an install and enabled lifecycle hook.

However, with Connect data residency migrations, neither realms will receive any of these lifecycle hooks as the app is neither uninstalled or installed.

During and prior to the migration, all hooks will continue to hit the current realm - and if the migration is successful, then subsequent hooks and webhooks will hit the target specified realm.

Hopefully that answers your question!

5

Hi @marc,

The data types considered in-scope for an app’s data residency will be dependent on a Partner’s/app’s individual privacy and data residency policy, which should be documented and made available to customers.

Atlassian has outlined details of what is in-scope for its products here.

Data in transit (up to 30 days), including data being processed and not at rest is currently not in-scope for Atlassian products data residency.

Thanks,
Sushant

6

Hi @SushantBista ,
another question: does the clientKey stay the same during and after migration? Currently the lifecycle hooks issue uninstall and install events, resulting in changed clientKeys (I believe).

And can we “query” a Confluence instance to get the current realm? The reason I’m asking is that we want to process data in the same realm as where the current Confluence instance is.

2 Likes
7

Hi @marc!

Just to quickly answer the question regarding the clientKey - yes, it will stay the same. In fact, irrespective of migrations, the clientKey should be persistent through installations and uninstallations. The only scenario a clientKey can change is during a site import which is outlined on this page.

If you do find the clientKey is changing for other reasons, please do reach out so we can explore further as that is not expected behaviour.

Thanks a lot!

2 Likes
8

Hi @marc, regarding your second question, currently there is no API to query the pinned location of the tenant/instance. Generally, consent would likely be required from the customer to provide such information to app developers. We are looking into the feasibility and options to provide such API to app developers (see update post). As we progress on this, we will provide an update on developer community.

Thanks,
Sushant

9

Hi @SushantBista ,
knowing the tenant location is really important for our customer and us. Some customers ask if we transfer data out of their current jurisdiction. If we know the tenant location, we can do the processing in the same jurisdiction, and so avoid transferring data out of the jurisdiction.
However if we don’t know the tenant location, we can tell our customers only “we don’t know if we transfer your data”, which will trigger all kinds of compliance related things.

2 Likes
10

Hi @SushantBista ,

thanks for the announcement! I took a look in the documentation and some questions came up while thinking the migration through:

  1. Which (Region Base) URL will receive the migration-hooks? Will it always be the URL of the “old” region, the URL of the “target” region, or will it always be the global region?
  2. Are these migration webhooks authenticated like the installed lifecycle events?
  3. What are the timeouts for the webhooks? Considering that the work done for the /commit webhook can also fail, it would be interesting to know how much time we have :slight_smile:

Cheers, Fabian

1 Like
11

Hi @FabianSiegel1!

To quickly answer some of your questions:

  1. It will always be the baseUrl of the currently installed app in the product. So essentially, the “old” region.
  2. So they won’t be authenticated like the installation lifecycle events since that utilises a different authentication mechanism (specifically signed-installs). But it does reuse the existing JWT authentication described on this page.
  3. While I’ll have to get back to you with the exact timeout - I can say it is definitely greater than 15s. Do you envision that you’d need more time than this?

Thanks a lot for reaching out!

1 Like
12

Thanks for your answers Jacky!

Maybe let me rephrase my question. From the current documentation I got the impression that when an app tells that it’s ready-to-commit, all data should already already be copied to the new region. So I assume with receiving the /commit hook, the old data should get deleted in the old region?

2 Likes
13

Hi, We have a connect app available in marketplace. I want to understand more about this program. I am completely new to this topic of Data residency.
I want to understand what is the program for ? What sort of app data is going to get migrated using this feature?
I understand the implementation part of it but I am bit confused about what exactly is getting migrated through this APIs.

Let me put my scenario here and can anyone explain in terms of it.

I have a Cloud Jira Site that I created. Also I installed a managed connect app on it. The app provides some additional integration features to Jira site but manages its data on its own servers let say on some cloud platform on AWS.

Now, I add the migration scope to my installed app as per the documentation. I want to understand what items of the app will be migrated as the app is managing the data somewhere on other servers.

We have supported Cloud Site Migrations for our app customers but did that with Lifecycle events where a site is renamed from AAA.atlassian.net to ZZZ.atlassian.net and with reinstall of app the existing app data is restored.

Some other questions:

  • Will all apps need to provide/implement this feature once it is production ready ?
  • App data residency - can anyone explain some scenarios ?

Thank you

6 Likes
14

Hi @PrerakDiwan!

Data residency for apps wrt to your scenario refers to any in-scope data stored by the app that is outside of the product is stored in a region which complies with the site’s data residency policy. For example, a customer located in Europe may be subject to regulations which require their data to be stored in Europe.

While the product may be pinned to Europe, and consequently the product data is stored in Europe. Apps may still store data which could be located anywhere. The intention of data residency realm pinning and migrations aims to address this. App developers can specify realms where their data is processed and stored in their descriptor. This subsequently has installations routed through to said realm if it matches the product’s residency. More can be read here.

The migration lifecycle declared here is to enable existing installations to migrate to one of your declared realms in the descriptor. So for example, a customer might be currently located in the Global realm - but has later pinned their product to the EU. To move the app server data across, a customer can trigger an app migration provided the apps support the desired realm. What is migrated is entirely up to you, we have no control over this as this is the app data stored on your server that should be migrating. Whatever is in-scope for your app is defined by your data residency policy and should be communicated to customers.

Finally, the newly introduced migration lifecycle is essentially an orchestration tool for an app to perform the migration whilst the site is offline.

15

Hiya @FabianSiegel1,

Sorry this flew past my radar. But yep that was the intention.

1 Like
16

To be honest this does not sound like a good idea. I would assume many vendors would opt-in for isolated databases for each location, so the customers will end up with the blank apps without any reasonable way to bring back the data.

3 Likes
17

A couple of questions regarding how should we manage locations:

  • what should we do if we decide to host EU data in Germany for example? Should we specify two locations pointing to the same thing, or only EU or DE?
  • as continutation of previous point, what happens if user wants to migrate data to DE, but we only support EU region or vice-versa?
  • in case when we specify EU and DE as the same url (it it’s the recommended approach), what happens when user requests a migration from EU to DE, or vice-versa?

Also a couple of technical questions (with assumption of isolated database for each location):

  • at which point the data from the source location should be removed?
  • is there a timeout after residency migration before user might attempt a new migration?
2 Likes
19

Hey @lexek-92,

Connect will treat DE and EU as seperate realms, as the declared EU realm could be hosted outside of Germany.

what should we do if we decide to host EU data in Germany for example? Should we specify two locations pointing to the same thing, or only EU or DE?

This will be up to app vendors to decide. Generally, if your EU realm is hosted in Germany, it will be ok to declare the same region for both. If your supported region for EU is NOT in Germany, then it cannot be used for DE declaration.

as continutation of previous point, what happens if user wants to migrate data to DE, but we only support EU region or vice-versa?
In case when we specify EU and DE as the same url (it it’s the recommended approach), what happens when user requests a migration from EU to DE, or vice-versa?

If your European DB is hosted in Germany, declaring the DE realm in your descriptor makes sense to ensure customers are aware that their in-scope is hosted in Germany.

If you only declare DE realm in the descriptor, the app will be shown as ‘ineligible’ to migrate to EU.

Optionally, for above scenario where your European DB is hosted in Germany, you could also declare both DE and EU in your descriptor, however from a Connect perspective, it would be treated as a seperate realms. There will be a few migration considerations to factor:

  • Connect will treat still EU and DE as seperate realms (as it could be pointing to different locations)
  • Customer would not know that the app is hosting in Germany for both EU and DE, and will assume a migration will take place if they request it
  • If migration is requested by customer (i.e. if host product also moves from EU to DE (or vice versa)), a migration schedule hook will still be sent to your app (assuming the relevant endpoints have been integrated)
  • However in above scenario, as both DE and EU are hosted in same DB in Germany, no actual migration will need to take place for the app, i.e. it can be set it up so the migration is commited to the new requested realm straight away during the migration window, without actually migrating any data
  • Once migration is complete, the customer will see an update in UI that the app pinning realm has changed e.g. from EU to DE.

in case when we specify EU and DE as the same url (it it’s the recommended approach), what happens when user requests a migration from EU to DE, or vice-versa?

There is no technical restriction for declaring the same URL for multiple declared realms in descriptor. There maybe a few reasons for you to choose to setup this way. One is the scenario you have outlined, where DE and EU both point to the same DB hosted in Germany. However apps are also likely to also use a single URL across multiple declared realms if they are re-routing to the region specific locations on the backend (e.g. via reverse-proxy etc.).

Again from a Connect perspective, if multiple realms are declared on descriptor, it will treat it as seperate locations (for realm pinning). Therefore if your app has also enabled realm migration capabilities, and the customer requests a migration from one supported realm to another, your app will get a migration request to migrate and commit to the requested realm (regardless of if an actual data migration takes place or not in the backend).

at which point the data from the source location should be removed?

Generally 30 days or less, but please let me confirm and get back to you on this.

is there a timeout after residency migration before user might attempt a new migration?

No, a ‘timeout’ has not been implemented at this stage.

Thanks,
Sushant

3 Likes
20

Thank you for clarifications.

So we should assume the possibility of situation when we might have old data retained (within the allowed period) in the target location when migration is triggered?
E.g customer migrated US → EU and a week later they decide to migrate EU → US

I also have a question in regard to foreceful re-pinning mentioned by @JackyLo in this thread:

Should we expect this logic to remain?
If yes:

  • what should we do with data in the old location when this happens? Should we apply regular retention policy for uninstalled apps in this case if customer is not actively asking for data restoration/migration?
  • should we delete old data (within data retention window, before it’s deleted according to policies) if user requests realm migration to old location?

E.g customer has product pinned to EU and app installed in global realm (let’s assume it’s US), they perform an uninstall and then re-install. They end up with the blank app state pinned to EU, but they decide not to pursure data migration with the app vendor. And week later they decide to migrate everything to US via realm migrator.

21

Hi @lexek-92,

at which point the data from the source location should be removed?

For Atlassian products, we endeavour to remove data from the source location as soon as possible after the realm migration process is complete. While we understand that you have a separate and independent relationship with your customers, we suggest that you also remove data from the source location as soon as possible after a successful migration. We also recommend that you specify your retention period for post-migration data in the source location in your product documentation, so your customers are aware.

So we should assume the possibility of situation when we might have old data retained (within the allowed period) in the target location when migration is triggered?

For Atlassian products, as the data from source location is attempted to be removed as soon as possible, the scenario you outlined will likely be treated as a seperate migration event.

Should we expect this logic to remain?

Yes. Connect will treat any installation as a fresh installation.

  • what should we do with data in the old location when this happens? Should we apply regular retention policy for uninstalled apps in this case if customer is not actively asking for data restoration/migration?
  • should we delete old data (within data retention window, before it’s deleted according to policies) if user requests realm migration to old location?

Generally, yes, if customer is not actively asking for data restoration/migration. As mentioned above, Connect regards any installation as a fresh installation.

Thanks,
Sushant

1 Like