Data residency realm migration for Connect apps: Early Access Program

Announcements
, ,
22

Hi @SushantBista,

Can you please elaborate on the reasoning behind this?

We’re 100% confident it will create a lot of issues for customers and vendors without any sanity checks. And as more vendors start supporting data residency, the issues will arise on a larger scale.

In fact, it’s already happening Data Residency : About when existing customer uninstall then install again

Some vendors, who might not be aware of this logic, are already starting to support data residency before any migration capabilities (even outside of Atlassian data residency migration platform) are implemented. Their customers are at risk of losing access to their data for a long periods of time, while vendors have to implement quick and barely tested (possibly semi-manual) migration schemes just to get data back to the customers ASAP.

23

Hi @SushantBista
We are stuck at the below scenario/behavior for existing customers as follows.

Customer Location AWS Region Region to which a Jira sends the request for add-on data
Please consider the current status as below (Before our add-on supports the Data Residency)
Customer_XYZ Globle EU-XX Environment before the Data Residency
Customer has some connect-app data in the environment before the Data Residency
Now we start supporting a Data Residency
Let’s look at the current status as follows
Customer_XYZ Globle EU-XX Environment before the Data Residency
Now the customer uninstalls the add-on and installs it again
Let’s look at the current status as follows
Customer_XYZ Globle EU-XX EU-XX
Customer lost/can not find the data as Jira now sending the request to the EU-XX/different server

Now the question is How can we tell Jira that for Customer_XYZ please send the request to the Globle environment?

NOTE: Customer did not enable the Data Residency for Jira and Add-on applications as well till this moment.

1 Like
24

Hi @CalendarJIRA

Just to confirm, is your concern regarding a scenario where customers have not requested a migration or have not been told to reinstall for purpose of moving the install, and instead uninstalled an app for other reasons and decision to reinstall again? For the case that it was a valid uninstallation scenario, I believe it would fall into standard uninstallation / data retention policy of the app.

Can you please elaborate on the reasoning behind this?

Connect treats all installations as a fresh install, as there could be valid reasons (outside of data residency) for an app to be uninstalled and possible reinstalled by customer in a later period. Generally, a reinstallation method is not the standard process for moving an app, however some apps maybe using it if they support some form of manual migration (i.e. they are able to connect the old data set to the new installation). This will require the customer to request such migration to the Partner.

Thanks,
Sushant

25

Hi, @SushantBista

Yes, this is our primary concern. Customers reinstall apps all the time, for troubleshooting or other reasons. And it’s not only about the cases when customers uninstall and then decide to reinstall after some time, it’s also about cases where customers explicitly want to uninstall and re-install app right away for whatever reason.

26

Hi @SushantBista ,

The problem with uninstall/reinstall is a hot mess and it needs to be Atlassian’s issue to fix before launching. I don’t see any reasonable way for individual vendors to deal with this without additional tooling from Atlassian.

Here is one way you could solve it:

  • The Connect installation realm needs to be sticky, per-app. Regardless of where the customer’s host site data is stored, Connect needs to remember the previous installation realm for each individual app (within, say, a 30-day window, or something reasonable that is well-documented). Reinstalls within that window must always use the previous realm. Your update from last November suggested that this was being considered. Are there any status updates?

  • Site admins need a dashboard displaying which realm is in use for all installed apps (ie. which install base)

  • For apps that have realms that differ from the host product, if the app supports migration to the realm of the host product, the dashboard should display a button that permits individual realm migration for that app (or, alternatively, for all apps). This must take into account that DE is a subset of EU (if the host application is pinned to DE, an app with data stored in US should be able to migrate to EU even if the app doesn’t support DE).

Failure to do all of this creates too many invisible problems and unexpected results. The transition of an app from one realm to another needs to be handled in a controlled manner, and an uninstall/reinstall forces an implicit and probably-unexpected realm switch. This creates horrible user experience issues, such as app data that disappears simply because somebody was troubleshooting.

Furthermore, the invisibility of the current app realm creates compliance holes for customers. Suppose that the Atlassian host application is pinned to Germany, but at the time of app realm migration, the Marketplace app only supported US. Six months later, the marketplace app now supports DE residency as well. The original sysadmin who did the migration has since left the company, so no history is available. When the client later looks at the current state of affairs (Jira says it’s hosted in DE and the app’s documentation says it supports DE hosting too), the reasonable assumption is that all of their data is hosted in DE and they are compliant, even though the app is an outlier with data stored in the US. There is no way for the customer to know this (so you have just created a compliance problem), nor is there any way for them to fix it.

2 Likes
27

Hi @scott.dudley, thank you for your feedback.

  • Site admins need a dashboard displaying which realm is in use for all installed apps (ie. which install base)
  • For apps that have realms that differ from the host product, if the app supports migration to the realm of the host product, the dashboard should display a button that permits individual realm migration for that app (or, alternatively, for all apps).

Both of these suggestions are being implemented. We are currently working on the customer experience which will allow admins (via AdminHub UI) to:

  • Display details of where the host product is pinned
  • Display list of installed apps (per site/host product) and their current pinned location
  • Status to show which of the installed apps are ‘eligible’ to migrate realm (in the case that the host product is in a different realm as installed apps). The eligibility of an app to migrate is determined based on if the relevant app supports the realm that the host product is pinned, as well as if the app supports automated migrations between its supported realms i.e. have integrated the realm migration APIs/migration hooks)
  • The admin will then be able to schedule a realm migration of ALL ‘eligible’ apps to be migrated to the same realm as the host product within a migration window scheduled by the admin

This must take into account that DE is a subset of EU (if the host application is pinned to DE, an app with data stored in US should be able to migrate to EU even if the app doesn’t support DE).

Regarding considerations between EU and DE realm declarations for the app, I have attempted to clarify in my above comment.

  • The Connect installation realm needs to be sticky, per-app. Regardless of where the customer’s host site data is stored, Connect needs to remember the previous installation realm for each individual app (within, say, a 30-day window, or something reasonable that is well-documented). Reinstalls within that window must always use the previous realm. Your update from last November suggested that this was being considered. Are there any status updates?

We appreciate the concern raised here. Realm persistence for re-installations is something we are still exploring to determine what is the best outcome for wider developer community.

Our understanding is that some apps are already/will continue doing some form of manual migration, and for this, these apps rely on the current uninstall/reinstall mechanism to move the installation location and then migrate a tenant’s data between realms. Therefore, we have not deprecated this mechanism yet.

Our current hypothesis is that once the realm migration solution is available to customers, most realm migrations can be facilitated using this new solution without requiring any reinstallation of the app (though there maybe edge cases where manual migrations are still required and apps continue to rely on existing uninstall/reinstall mechanism). Therefore, most subsequent un-installations/re-installations would likely be performed by customers for non data-residency reasons, e.g. troubleshooting as you outlined or a change of mind. However, this may be an issue if the host product has been migrated to a new realm during this uninstallation/reinstallation process → where Connect will treat it as a fresh installation pinned to the same realm as host product, and will not account for previous data set in old realm. Our assumption here is this scenario would be relatively low occurrence once realm migration capability is made available to customers. Further, this occurrence rate would continue to decrease as more apps integrate the realm migration solution. What are your thoughts on this assumption?

Another potential scenario which may cause issues is if a customer installed an app prior to adoption of realm pinning/relevant realms by the app, then the customer uninstalls and reinstalls, and the app traffic is now routed to the new realm (same as host product), and causing a disconnect with the old data set in previous realm. This is something we are looking into further to understand how we can address it.

Please note, If we were to implement realm persistence upon reinstall in future (TBD), it could result in a breaking change for apps utilising current uninstall/reinstall mechanism, and therefore, we believe should only be implemented once the official realm migration solution is GA and majority of apps have implemented the solution. This will also require a deprecation notice/period prior to implementation.

When the client later looks at the current state of affairs (Jira says it’s hosted in DE and the app’s documentation says it supports DE hosting too), the reasonable assumption is that all of their data is hosted in DE and they are compliant, even though the app is an outlier with data stored in the US. There is no way for the customer to know this (so you have just created a compliance problem), nor is there any way for them to fix it.

As outlined above, the upcoming Admin UI should address this issue as it will outline the installed app’s current pinned realm, which in your example above will show differs from the host products pinned realm. It will also provide ability for the admin to then migrate the ‘eligible’ apps to be in the same realm as the host product.

Thanks,
Sushant

28

It’s hard for me to tell how widely it’s used for this purpose. But looking from a developer perspective, this poses to do more harm than good for customers without a way to control this.

The current approach will require for developers to implement realm migrations both via realm migration platform and outside of it, creating a considerable amount of additional work.

It also creates an implicit requirement for vendors to support realm migration in order to support data residency for new customers. Because there is always a risk for some customer to accidentally re-pin their installation into other realm outside of data residency realm migration.

As a reasonable compromise, would it be possible to introduce some kind of field or apiMigrations toggle in atlassian-connect.json descriptor to enforce realm persistence for apps that only support realm change via realm migration platform? Or, a toggle to allow re-pinning via reinstallation in app descriptor.

1 Like
29

Hi @SushantBista,

Thanks for the detailed update. I appreciate all of the effort that has gone towards solving the issues with the dashboard and permitting ad-hoc realm migration for apps.

With regards to the Connect installation realm being sticky, I also appreciate the additional information about the other use cases for it being non-sticky. However, I do believe that enforcing the prioritization the way you suggest would do more harm than good.

As I understand it now, there are two separate uses cases for the uninstall/reinstall lifecycle:

  1. The app vendor is doing a one-time manual realm migration and they are using the uninstall/reinstall process to switch the realm of the app.
  2. The app is being uninstalled for any other reason, including troubleshooting, maintenance, expiration of trial, and a myriad of other possibilities.

Manual realm migrations in #1 are very low-occurrence. Other reinstalls for #2 are, in relative terms, very high-occurrence.

If this uninstall/reinstall occurs at any time after a realm migration is performed, but before the app supported migration to that realm, this creates an issue. I do not think that #2 necessarily becomes “relatively low occurrence once realm migration capability is made available to customers”: even if the app adds support for realm migration at a later date (compared to the pinning of the host product), I don’t think we can count on admins regularly going to the dashboard to migrate additional apps. So, even if you have 100% of apps supporting realm migrations, you will still have this problem.

Realm switching is an important event, and having it happen in an uncontrolled and unexpected manner is unquestionably bad. But with #2, this is exactly what happens. The magnitude of the issue created by this problem is also huge (all of the customer’s app data is unexpectedly gone, and even worse, the only way to fix it creates heaps of work for the app vendor!).

#1 is a low-frequency, high-touch event that, by definition, already requires manual involvement of support personnel from the vendor. It seems that this case is more of the exception than the rule, and if it took a manual ticket raised with Atlassian to unpin the app realm, I think this would be entirely reasonable. If this adds two extra days to the process to unpin the app, how many people care? Or, alternatively, allow apps to specify their preference through the app descriptor, as lexek-92 suggested.

In summary, causing #2 to break regularly with catastrophic results, in order to help prevent breaking #1, feels like the wrong trade-off. Especially since the only people doing manual realm migrations with #1 are using a legacy process, who you probably want to push to use the automated realm migration solution anyway!

This must take into account that DE is a subset of EU (if the host application is pinned to DE, an app with data stored in US should be able to migrate to EU even if the app doesn’t support DE).

Regarding considerations between EU and DE realm declarations for the app, I have attempted to clarify in my above comment.

It seems odd that, if an app supports EU data residency but not DE specifically, then a DE-pinned customer will not be able to migrate existing data out of whatever location may be in use (let’s say, the US) to at least bring it into the EU.

Scott

2 Likes
30

As a potential solution, Atlassian might implement Data Residency preference as prioritized list instead of single choice for customer, where they can pick acceptable data residence options.
So in this case the customer might pick DE as first choice and EU as second acceptable realm which will be used in cases when DE is not available

31

Hey @lexek-92, thank you for the feedback. Definitely acknowledging the valid concerns you have raised.

As a reasonable compromise, would it be possible to introduce some kind of field or apiMigrations toggle in atlassian-connect.json descriptor to enforce realm persistence for apps that only support realm change via realm migration platform? Or, a toggle to allow re-pinning via reinstallation in app descriptor.

The team will assess this option you have outlined. Thank you for suggesting this. Will try and keep you in the loop.

Thanks,
Sushant

32

Hi @scott.dudley, really appreciate the detailed feedback you have provided.

In summary, causing #2 to break regularly with catastrophic results, in order to help prevent breaking #1, feels like the wrong trade-off. Especially since the only people doing manual realm migrations with #1 are using a legacy process, who you probably want to push to use the automated realm migration solution anyway!

This is something we are looking into more closely. We will factor in your feedback as we assess this further. Will definitely keep you in the loop as we progress.

As a potential solution, Atlassian might implement Data Residency preference as prioritized list instead of single choice for customer, where they can pick acceptable data residence options.
So in this case the customer might pick DE as first choice and EU as second acceptable realm which will be used in cases when DE is not available

@lexek-92 thanks for the suggestion. Will discuss this with the team and get back to you both.

Thanks,
Sushant

33

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.