Seeking feedback: Realm to realm migration for Connect Apps

Let’s talk migrations

As we talked about in this post, the most complicated part about supporting data residency will be supporting a realm to realm migration. During a migration, as Atlassian migrates the product’s data, Connect will be unavailable for some portion of the migration process. Our migration flows we are testing reflect this:

The product plus app migration process

  1. Connect messages app that the user has scheduled a migration (Endpoint 1)
  2. Connect messages the app for a status if it’s ready to migrate (Endpoint 3)
  3. Connect messages app that the scheduled migration is commencing and tenantis unavailable (Endpoint 2)
  4. Connect messages app to ask if done migrating (Endpoint 3)
  5. If App says we have not completed
    1. Connect points to previous region
    2. Atlassian signals to user that apps was unsuccessful via email/support/UI
    3. Atlassian provides process to schedule ​ app-specific migration in the future via Adminhub/Support
  6. If App says we have completed
    1. Connect points to new region
  7. Connect says commit (Endpoint 4)

The app-specific migration process

  1. Connect messages app that the user has scheduled a migration (Endpoint 1)
  2. Connect messages the app for a status if it ready to migrate (Endpoint 3)
  3. Connect messages app that the scheduled migration is commencing and tenantis available (Endpoint 2)
  4. Connect has a pop up signaling that an app migration is in progress and thatthere is limited app functionality
  5. Connect messages app to ask if done migrating (Endpoint 3)
  6. If App says we have not completed
    1. Connect points to previous region
    2. Atlassian signals to user that apps were unsuccessful via email/support/UI
    3. Atlassian provides process to schedule app specific migration in the future via Adminhub/Support
  7. If App says we have completed
    1. Connect points to new region
    2. Atlassian signals migration is complete to users
  8. Connect says commit (Endpoint 4)

UI​ : ​ Atlassian will look to support the admin experience by creating an app data residency management view.

What will our endpoints look like?

Endpoint Function Response
Endpoint 1: /migration/schedule Tells the app of the time that the admin has booked into to migrate 2xx to indicate the migration has been accepted or, 4xx to indicate the app is rejecting the migration
Endpoint 2: /migration/start Tells the app that Atlassian wishes to migrate the app data to a new region. The addon should prevent any modifications to data in the site’s current region. The app data for the tenant should be copied to the new region 2xx to indicate the migration has been accepted and the tenant is offline or 4xx to indicate the app is rejecting the migration
Endpoint 3: /migration/status Indicates the status of the migration Migrations can be: Complete, In-progress with percentage or Failed
Endpoint 4: /migration/commit Indicates that Atlassian is ready to commit the site to the new region. The app should enable the site in the new region it would be considered illegal for an app to fail this operation if it has indicated the migration status is complete Failure at this point would require manual intervention

Please provide us feedback on the design in the form of comments on this post.

3 Likes

Hey!

First gut reaction is this sounds good. The link in the first paragraph doesn’t work for me though, as I don’t have access to hello.atlassian.net (should I or is it Atlassian-internal?).

Have a few questions / remarks though:

  • If I read this correctly, at no point will the /installed webhook be called, so I’m assuming the key material Connect has established with our app for that instance does not change? And neither will the app’s domain?
  • I’m also assuming we don’t have to take care of migrating macro, entity or app properties, as they are stored in the host product, correct?
  • Would it be possible to include a (somewhat generalized) data transfer mechanism in atlassian-connect-express so we don’t all have to duplicate the code for doing this? I’m assuming Atlassian has to do this for their Connect apps as well
  • Is there a rough timeline for testing this in Connect apps with dev instances and subsequent public release of the feature?
  • (minor oversight, I think) Endpoint 3 does not currently seem to have a response for product + app migration scenario step 2
8 Likes

@tobitheo, I’m guessing the post @valente is referring to is this one.

2 Likes

Looking forward to see data residency and migration to be supported in atlassian-connect-express!

What currently seems to be missing from the proposal is the handling of user data. Are marketplace partners supposed to apply data residency to user data, and if so, how?

2 Likes

Another +1 for a reference implementation in ACE.

5 Likes

+1 for a reference implementation in ACE.

And a question: is the app supposed to transfer the tenant registration info it’s holding for the Jira instance (and which is currently managed by ACE)? Is that considered as “app data”? Or will the /installed webhook be called on the new region?

Also, the backend-to-backend communication between regions will require strong security. Is that also something that ACE could provide?

2 Likes

We are currently working on Cloud migrations, and for Cloud migrations there is a map of old/new issue IDs, user IDs, etc. So my question: can we assume that these IDs will remain the same on the target environment, as this is essentially a lift and shift migration?

Hey Marc.

Good question. We leave what types of data that are covered by the partner completely up to the marketplace partner. They can specify this in their data residency policy. Atlassian’s policy does not currently cover user data.

Hey Vitor,

Yep this is a lift and shift. You can assume all ids remain the same.

2 Likes

Thanks everyone for the recommendation to put a reference implementation into ACE. We will look to make reference implementation a part of our rollout of the migration support.

2 Likes

Hey David,

What is considered to be covered by your data residency policy is up to you. You can see what Atlassian covers in its data residency policy here
As for your comment about backend-to backend communication security protocols, that has been duly noted and we will endeavour to provide an example in reference docs.

Yet another +1 for a reference implementation in ACE. (And also thank you for picking that up.)

Plus one more question: Will there be a certain timeframe within which a migration needs to be completed? From what I understand, Connect will allow ask the App once whether it is done migrating and then either points to the new region or escalates an error.

3 Likes

Hi @valente,

What is considered to be covered by your data residency policy is up to you. You can see what Atlassian covers in its data residency policy here

I’m not sure how this is related to my question… My question was:

is the app supposed to transfer the tenant registration info it’s holding for the Jira instance (and which is currently managed by ACE)? Is that considered as “app data”? Or will the /installed webhook be called on the new region?

Thanks,
David

2 Likes

Hey David,

We’re going to call the /install webhook in the new region. It will be an app responsibility to connect or transfer to any existing data they have kept in the old region over to the new region.

1 Like

@valente

Do we need to uninstall the app from the old region?

I assume then that the install hook in the new region will be called with the previous secret so that the installation is verified? This then means that the sharedSecret’s and tenant installations need to be stored outside of the geolocated data?

Will the /install hook be called the moment that the admin selects the location?

Would really love to see all of this implemented in ACE because this is going to become complicated really quickly.

2 Likes

We’re assuming that we migrate the secret as part of the app migration, which could also work. But yes, great question and would love to learn what is expected / best practice here.

Is there any kind of progress in this space. We have the first app live with data residency and following up with the migration path would seem the next logical step. I hope to avoid one-off migration requests.

4 Likes

Hi Everyone,

We are currently working out the specifications of app migrations. This is something that we are prioritising and will update you once we have more clarity on migration path.

In the meantime, please let me know if you have any particular concerns or questions regarding migrations.

1 Like

why do we need a commit notification specifically ? since Endpoint 3 already indicates if “Complete” or not