Data residency realm migration for Connect apps: Early Access Program

Announcements
, ,
13

Hi, We have a connect app available in marketplace. I want to understand more about this program. I am completely new to this topic of Data residency.
I want to understand what is the program for ? What sort of app data is going to get migrated using this feature?
I understand the implementation part of it but I am bit confused about what exactly is getting migrated through this APIs.

Let me put my scenario here and can anyone explain in terms of it.

I have a Cloud Jira Site that I created. Also I installed a managed connect app on it. The app provides some additional integration features to Jira site but manages its data on its own servers let say on some cloud platform on AWS.

Now, I add the migration scope to my installed app as per the documentation. I want to understand what items of the app will be migrated as the app is managing the data somewhere on other servers.

We have supported Cloud Site Migrations for our app customers but did that with Lifecycle events where a site is renamed from AAA.atlassian.net to ZZZ.atlassian.net and with reinstall of app the existing app data is restored.

Some other questions:

  • Will all apps need to provide/implement this feature once it is production ready ?
  • App data residency - can anyone explain some scenarios ?

Thank you

8 Likes
14

Hi @PrerakDiwan!

Data residency for apps wrt to your scenario refers to any in-scope data stored by the app that is outside of the product is stored in a region which complies with the site’s data residency policy. For example, a customer located in Europe may be subject to regulations which require their data to be stored in Europe.

While the product may be pinned to Europe, and consequently the product data is stored in Europe. Apps may still store data which could be located anywhere. The intention of data residency realm pinning and migrations aims to address this. App developers can specify realms where their data is processed and stored in their descriptor. This subsequently has installations routed through to said realm if it matches the product’s residency. More can be read here.

The migration lifecycle declared here is to enable existing installations to migrate to one of your declared realms in the descriptor. So for example, a customer might be currently located in the Global realm - but has later pinned their product to the EU. To move the app server data across, a customer can trigger an app migration provided the apps support the desired realm. What is migrated is entirely up to you, we have no control over this as this is the app data stored on your server that should be migrating. Whatever is in-scope for your app is defined by your data residency policy and should be communicated to customers.

Finally, the newly introduced migration lifecycle is essentially an orchestration tool for an app to perform the migration whilst the site is offline.

15

Hiya @FabianSiegel1,

Sorry this flew past my radar. But yep that was the intention.

1 Like
16

To be honest this does not sound like a good idea. I would assume many vendors would opt-in for isolated databases for each location, so the customers will end up with the blank apps without any reasonable way to bring back the data.

3 Likes
17

A couple of questions regarding how should we manage locations:

  • what should we do if we decide to host EU data in Germany for example? Should we specify two locations pointing to the same thing, or only EU or DE?
  • as continutation of previous point, what happens if user wants to migrate data to DE, but we only support EU region or vice-versa?
  • in case when we specify EU and DE as the same url (it it’s the recommended approach), what happens when user requests a migration from EU to DE, or vice-versa?

Also a couple of technical questions (with assumption of isolated database for each location):

  • at which point the data from the source location should be removed?
  • is there a timeout after residency migration before user might attempt a new migration?
2 Likes
19

Hey @lexek-92,

Connect will treat DE and EU as seperate realms, as the declared EU realm could be hosted outside of Germany.

what should we do if we decide to host EU data in Germany for example? Should we specify two locations pointing to the same thing, or only EU or DE?

This will be up to app vendors to decide. Generally, if your EU realm is hosted in Germany, it will be ok to declare the same region for both. If your supported region for EU is NOT in Germany, then it cannot be used for DE declaration.

as continutation of previous point, what happens if user wants to migrate data to DE, but we only support EU region or vice-versa?
In case when we specify EU and DE as the same url (it it’s the recommended approach), what happens when user requests a migration from EU to DE, or vice-versa?

If your European DB is hosted in Germany, declaring the DE realm in your descriptor makes sense to ensure customers are aware that their in-scope is hosted in Germany.

If you only declare DE realm in the descriptor, the app will be shown as ‘ineligible’ to migrate to EU.

Optionally, for above scenario where your European DB is hosted in Germany, you could also declare both DE and EU in your descriptor, however from a Connect perspective, it would be treated as a seperate realms. There will be a few migration considerations to factor:

  • Connect will treat still EU and DE as seperate realms (as it could be pointing to different locations)
  • Customer would not know that the app is hosting in Germany for both EU and DE, and will assume a migration will take place if they request it
  • If migration is requested by customer (i.e. if host product also moves from EU to DE (or vice versa)), a migration schedule hook will still be sent to your app (assuming the relevant endpoints have been integrated)
  • However in above scenario, as both DE and EU are hosted in same DB in Germany, no actual migration will need to take place for the app, i.e. it can be set it up so the migration is commited to the new requested realm straight away during the migration window, without actually migrating any data
  • Once migration is complete, the customer will see an update in UI that the app pinning realm has changed e.g. from EU to DE.

in case when we specify EU and DE as the same url (it it’s the recommended approach), what happens when user requests a migration from EU to DE, or vice-versa?

There is no technical restriction for declaring the same URL for multiple declared realms in descriptor. There maybe a few reasons for you to choose to setup this way. One is the scenario you have outlined, where DE and EU both point to the same DB hosted in Germany. However apps are also likely to also use a single URL across multiple declared realms if they are re-routing to the region specific locations on the backend (e.g. via reverse-proxy etc.).

Again from a Connect perspective, if multiple realms are declared on descriptor, it will treat it as seperate locations (for realm pinning). Therefore if your app has also enabled realm migration capabilities, and the customer requests a migration from one supported realm to another, your app will get a migration request to migrate and commit to the requested realm (regardless of if an actual data migration takes place or not in the backend).

at which point the data from the source location should be removed?

Generally 30 days or less, but please let me confirm and get back to you on this.

is there a timeout after residency migration before user might attempt a new migration?

No, a ‘timeout’ has not been implemented at this stage.

Thanks,
Sushant

3 Likes
20

Thank you for clarifications.

So we should assume the possibility of situation when we might have old data retained (within the allowed period) in the target location when migration is triggered?
E.g customer migrated US → EU and a week later they decide to migrate EU → US

I also have a question in regard to foreceful re-pinning mentioned by @JackyLo in this thread:

Should we expect this logic to remain?
If yes:

  • what should we do with data in the old location when this happens? Should we apply regular retention policy for uninstalled apps in this case if customer is not actively asking for data restoration/migration?
  • should we delete old data (within data retention window, before it’s deleted according to policies) if user requests realm migration to old location?

E.g customer has product pinned to EU and app installed in global realm (let’s assume it’s US), they perform an uninstall and then re-install. They end up with the blank app state pinned to EU, but they decide not to pursure data migration with the app vendor. And week later they decide to migrate everything to US via realm migrator.

21

Hi @lexek-92,

at which point the data from the source location should be removed?

For Atlassian products, we endeavour to remove data from the source location as soon as possible after the realm migration process is complete. While we understand that you have a separate and independent relationship with your customers, we suggest that you also remove data from the source location as soon as possible after a successful migration. We also recommend that you specify your retention period for post-migration data in the source location in your product documentation, so your customers are aware.

So we should assume the possibility of situation when we might have old data retained (within the allowed period) in the target location when migration is triggered?

For Atlassian products, as the data from source location is attempted to be removed as soon as possible, the scenario you outlined will likely be treated as a seperate migration event.

Should we expect this logic to remain?

Yes. Connect will treat any installation as a fresh installation.

  • what should we do with data in the old location when this happens? Should we apply regular retention policy for uninstalled apps in this case if customer is not actively asking for data restoration/migration?
  • should we delete old data (within data retention window, before it’s deleted according to policies) if user requests realm migration to old location?

Generally, yes, if customer is not actively asking for data restoration/migration. As mentioned above, Connect regards any installation as a fresh installation.

Thanks,
Sushant

1 Like
22

Hi @SushantBista,

Can you please elaborate on the reasoning behind this?

We’re 100% confident it will create a lot of issues for customers and vendors without any sanity checks. And as more vendors start supporting data residency, the issues will arise on a larger scale.

In fact, it’s already happening Data Residency : About when existing customer uninstall then install again

Some vendors, who might not be aware of this logic, are already starting to support data residency before any migration capabilities (even outside of Atlassian data residency migration platform) are implemented. Their customers are at risk of losing access to their data for a long periods of time, while vendors have to implement quick and barely tested (possibly semi-manual) migration schemes just to get data back to the customers ASAP.

23

Hi @SushantBista
We are stuck at the below scenario/behavior for existing customers as follows.

Customer Location AWS Region Region to which a Jira sends the request for add-on data
Please consider the current status as below (Before our add-on supports the Data Residency)
Customer_XYZ Globle EU-XX Environment before the Data Residency
Customer has some connect-app data in the environment before the Data Residency
Now we start supporting a Data Residency
Let’s look at the current status as follows
Customer_XYZ Globle EU-XX Environment before the Data Residency
Now the customer uninstalls the add-on and installs it again
Let’s look at the current status as follows
Customer_XYZ Globle EU-XX EU-XX
Customer lost/can not find the data as Jira now sending the request to the EU-XX/different server

Now the question is How can we tell Jira that for Customer_XYZ please send the request to the Globle environment?

NOTE: Customer did not enable the Data Residency for Jira and Add-on applications as well till this moment.

1 Like
24

Hi @CalendarJIRA

Just to confirm, is your concern regarding a scenario where customers have not requested a migration or have not been told to reinstall for purpose of moving the install, and instead uninstalled an app for other reasons and decision to reinstall again? For the case that it was a valid uninstallation scenario, I believe it would fall into standard uninstallation / data retention policy of the app.

Can you please elaborate on the reasoning behind this?

Connect treats all installations as a fresh install, as there could be valid reasons (outside of data residency) for an app to be uninstalled and possible reinstalled by customer in a later period. Generally, a reinstallation method is not the standard process for moving an app, however some apps maybe using it if they support some form of manual migration (i.e. they are able to connect the old data set to the new installation). This will require the customer to request such migration to the Partner.

Thanks,
Sushant

25

Hi, @SushantBista

Yes, this is our primary concern. Customers reinstall apps all the time, for troubleshooting or other reasons. And it’s not only about the cases when customers uninstall and then decide to reinstall after some time, it’s also about cases where customers explicitly want to uninstall and re-install app right away for whatever reason.

26

Hi @SushantBista ,

The problem with uninstall/reinstall is a hot mess and it needs to be Atlassian’s issue to fix before launching. I don’t see any reasonable way for individual vendors to deal with this without additional tooling from Atlassian.

Here is one way you could solve it:

  • The Connect installation realm needs to be sticky, per-app. Regardless of where the customer’s host site data is stored, Connect needs to remember the previous installation realm for each individual app (within, say, a 30-day window, or something reasonable that is well-documented). Reinstalls within that window must always use the previous realm. Your update from last November suggested that this was being considered. Are there any status updates?

  • Site admins need a dashboard displaying which realm is in use for all installed apps (ie. which install base)

  • For apps that have realms that differ from the host product, if the app supports migration to the realm of the host product, the dashboard should display a button that permits individual realm migration for that app (or, alternatively, for all apps). This must take into account that DE is a subset of EU (if the host application is pinned to DE, an app with data stored in US should be able to migrate to EU even if the app doesn’t support DE).

Failure to do all of this creates too many invisible problems and unexpected results. The transition of an app from one realm to another needs to be handled in a controlled manner, and an uninstall/reinstall forces an implicit and probably-unexpected realm switch. This creates horrible user experience issues, such as app data that disappears simply because somebody was troubleshooting.

Furthermore, the invisibility of the current app realm creates compliance holes for customers. Suppose that the Atlassian host application is pinned to Germany, but at the time of app realm migration, the Marketplace app only supported US. Six months later, the marketplace app now supports DE residency as well. The original sysadmin who did the migration has since left the company, so no history is available. When the client later looks at the current state of affairs (Jira says it’s hosted in DE and the app’s documentation says it supports DE hosting too), the reasonable assumption is that all of their data is hosted in DE and they are compliant, even though the app is an outlier with data stored in the US. There is no way for the customer to know this (so you have just created a compliance problem), nor is there any way for them to fix it.

2 Likes
27

Hi @scott.dudley, thank you for your feedback.

  • Site admins need a dashboard displaying which realm is in use for all installed apps (ie. which install base)
  • For apps that have realms that differ from the host product, if the app supports migration to the realm of the host product, the dashboard should display a button that permits individual realm migration for that app (or, alternatively, for all apps).

Both of these suggestions are being implemented. We are currently working on the customer experience which will allow admins (via AdminHub UI) to:

  • Display details of where the host product is pinned
  • Display list of installed apps (per site/host product) and their current pinned location
  • Status to show which of the installed apps are ‘eligible’ to migrate realm (in the case that the host product is in a different realm as installed apps). The eligibility of an app to migrate is determined based on if the relevant app supports the realm that the host product is pinned, as well as if the app supports automated migrations between its supported realms i.e. have integrated the realm migration APIs/migration hooks)
  • The admin will then be able to schedule a realm migration of ALL ‘eligible’ apps to be migrated to the same realm as the host product within a migration window scheduled by the admin

This must take into account that DE is a subset of EU (if the host application is pinned to DE, an app with data stored in US should be able to migrate to EU even if the app doesn’t support DE).

Regarding considerations between EU and DE realm declarations for the app, I have attempted to clarify in my above comment.

  • The Connect installation realm needs to be sticky, per-app. Regardless of where the customer’s host site data is stored, Connect needs to remember the previous installation realm for each individual app (within, say, a 30-day window, or something reasonable that is well-documented). Reinstalls within that window must always use the previous realm. Your update from last November suggested that this was being considered. Are there any status updates?

We appreciate the concern raised here. Realm persistence for re-installations is something we are still exploring to determine what is the best outcome for wider developer community.

Our understanding is that some apps are already/will continue doing some form of manual migration, and for this, these apps rely on the current uninstall/reinstall mechanism to move the installation location and then migrate a tenant’s data between realms. Therefore, we have not deprecated this mechanism yet.

Our current hypothesis is that once the realm migration solution is available to customers, most realm migrations can be facilitated using this new solution without requiring any reinstallation of the app (though there maybe edge cases where manual migrations are still required and apps continue to rely on existing uninstall/reinstall mechanism). Therefore, most subsequent un-installations/re-installations would likely be performed by customers for non data-residency reasons, e.g. troubleshooting as you outlined or a change of mind. However, this may be an issue if the host product has been migrated to a new realm during this uninstallation/reinstallation process → where Connect will treat it as a fresh installation pinned to the same realm as host product, and will not account for previous data set in old realm. Our assumption here is this scenario would be relatively low occurrence once realm migration capability is made available to customers. Further, this occurrence rate would continue to decrease as more apps integrate the realm migration solution. What are your thoughts on this assumption?

Another potential scenario which may cause issues is if a customer installed an app prior to adoption of realm pinning/relevant realms by the app, then the customer uninstalls and reinstalls, and the app traffic is now routed to the new realm (same as host product), and causing a disconnect with the old data set in previous realm. This is something we are looking into further to understand how we can address it.

Please note, If we were to implement realm persistence upon reinstall in future (TBD), it could result in a breaking change for apps utilising current uninstall/reinstall mechanism, and therefore, we believe should only be implemented once the official realm migration solution is GA and majority of apps have implemented the solution. This will also require a deprecation notice/period prior to implementation.

When the client later looks at the current state of affairs (Jira says it’s hosted in DE and the app’s documentation says it supports DE hosting too), the reasonable assumption is that all of their data is hosted in DE and they are compliant, even though the app is an outlier with data stored in the US. There is no way for the customer to know this (so you have just created a compliance problem), nor is there any way for them to fix it.

As outlined above, the upcoming Admin UI should address this issue as it will outline the installed app’s current pinned realm, which in your example above will show differs from the host products pinned realm. It will also provide ability for the admin to then migrate the ‘eligible’ apps to be in the same realm as the host product.

Thanks,
Sushant

28

It’s hard for me to tell how widely it’s used for this purpose. But looking from a developer perspective, this poses to do more harm than good for customers without a way to control this.

The current approach will require for developers to implement realm migrations both via realm migration platform and outside of it, creating a considerable amount of additional work.

It also creates an implicit requirement for vendors to support realm migration in order to support data residency for new customers. Because there is always a risk for some customer to accidentally re-pin their installation into other realm outside of data residency realm migration.

As a reasonable compromise, would it be possible to introduce some kind of field or apiMigrations toggle in atlassian-connect.json descriptor to enforce realm persistence for apps that only support realm change via realm migration platform? Or, a toggle to allow re-pinning via reinstallation in app descriptor.

1 Like
29

Hi @SushantBista,

Thanks for the detailed update. I appreciate all of the effort that has gone towards solving the issues with the dashboard and permitting ad-hoc realm migration for apps.

With regards to the Connect installation realm being sticky, I also appreciate the additional information about the other use cases for it being non-sticky. However, I do believe that enforcing the prioritization the way you suggest would do more harm than good.

As I understand it now, there are two separate uses cases for the uninstall/reinstall lifecycle:

  1. The app vendor is doing a one-time manual realm migration and they are using the uninstall/reinstall process to switch the realm of the app.
  2. The app is being uninstalled for any other reason, including troubleshooting, maintenance, expiration of trial, and a myriad of other possibilities.

Manual realm migrations in #1 are very low-occurrence. Other reinstalls for #2 are, in relative terms, very high-occurrence.

If this uninstall/reinstall occurs at any time after a realm migration is performed, but before the app supported migration to that realm, this creates an issue. I do not think that #2 necessarily becomes “relatively low occurrence once realm migration capability is made available to customers”: even if the app adds support for realm migration at a later date (compared to the pinning of the host product), I don’t think we can count on admins regularly going to the dashboard to migrate additional apps. So, even if you have 100% of apps supporting realm migrations, you will still have this problem.

Realm switching is an important event, and having it happen in an uncontrolled and unexpected manner is unquestionably bad. But with #2, this is exactly what happens. The magnitude of the issue created by this problem is also huge (all of the customer’s app data is unexpectedly gone, and even worse, the only way to fix it creates heaps of work for the app vendor!).

#1 is a low-frequency, high-touch event that, by definition, already requires manual involvement of support personnel from the vendor. It seems that this case is more of the exception than the rule, and if it took a manual ticket raised with Atlassian to unpin the app realm, I think this would be entirely reasonable. If this adds two extra days to the process to unpin the app, how many people care? Or, alternatively, allow apps to specify their preference through the app descriptor, as lexek-92 suggested.

In summary, causing #2 to break regularly with catastrophic results, in order to help prevent breaking #1, feels like the wrong trade-off. Especially since the only people doing manual realm migrations with #1 are using a legacy process, who you probably want to push to use the automated realm migration solution anyway!

This must take into account that DE is a subset of EU (if the host application is pinned to DE, an app with data stored in US should be able to migrate to EU even if the app doesn’t support DE).

Regarding considerations between EU and DE realm declarations for the app, I have attempted to clarify in my above comment.

It seems odd that, if an app supports EU data residency but not DE specifically, then a DE-pinned customer will not be able to migrate existing data out of whatever location may be in use (let’s say, the US) to at least bring it into the EU.

Scott

2 Likes
30

As a potential solution, Atlassian might implement Data Residency preference as prioritized list instead of single choice for customer, where they can pick acceptable data residence options.
So in this case the customer might pick DE as first choice and EU as second acceptable realm which will be used in cases when DE is not available

31

Hey @lexek-92, thank you for the feedback. Definitely acknowledging the valid concerns you have raised.

As a reasonable compromise, would it be possible to introduce some kind of field or apiMigrations toggle in atlassian-connect.json descriptor to enforce realm persistence for apps that only support realm change via realm migration platform? Or, a toggle to allow re-pinning via reinstallation in app descriptor.

The team will assess this option you have outlined. Thank you for suggesting this. Will try and keep you in the loop.

Thanks,
Sushant

32

Hi @scott.dudley, really appreciate the detailed feedback you have provided.

In summary, causing #2 to break regularly with catastrophic results, in order to help prevent breaking #1, feels like the wrong trade-off. Especially since the only people doing manual realm migrations with #1 are using a legacy process, who you probably want to push to use the automated realm migration solution anyway!

This is something we are looking into more closely. We will factor in your feedback as we assess this further. Will definitely keep you in the loop as we progress.

As a potential solution, Atlassian might implement Data Residency preference as prioritized list instead of single choice for customer, where they can pick acceptable data residence options.
So in this case the customer might pick DE as first choice and EU as second acceptable realm which will be used in cases when DE is not available

@lexek-92 thanks for the suggestion. Will discuss this with the team and get back to you both.

Thanks,
Sushant

33

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.