Update on data residency migrations for Connect apps

Hi, I’m Jacky, an engineer at Atlassian. Our team is working on data residency migrations for Connect apps. Back in 2020, we sought out some feedback on realm-to-realm migrations for Connect apps in the Atlassian Developer Community. We’re now looking to provide you with the following:

  • some assumptions we’re making,
  • more details on the migration endpoints we’ve been working on,
  • and an update on the app migration process.

Note, this is only a preview of the functionality we’re building, and is not a confirmation of its final implementation. We’ll provide official documentation of the functionality once it’s fully released.

Assumptions

Below are the assumptions we’re making:

  • App migrations will be done while the site is down.
  • No apps will be uninstalled as a result of the site migration process.
  • App migrations will be a non-destructive copy operation.
  • App migrations will not block site migrations.
  • App migrations will be performed at a site-level.
  • The eligibility of an app to be migrated will be decided at the scheduling time.

Revisiting the app migration process

In our previous post in the Atlassian Developer Community, we briefly described the realm-to-realm migration process that we envisioned at the time. With the above assumptions, the process has more or less stayed the same, with the exception of the following minor changes:

  • A new service will be introduced to orchestrate the migration process for apps. The service will continue to use JWT auth — signed with the sharedSecret you’ve obtained in the installed payload.
  • A rollback hook will be introduced to notify apps that the migration is not to be processed and the original copy of the data will be used.
  • The status hook serves a slightly different role. More details about this below.
  • Custom error response codes have been introduced to help diagnose failures.

With these updates, the process will now be as follows:

  1. The product admin will schedule a migration of eligible apps.
  2. The new service will notify eligible apps via the schedule endpoint (endpoint 1).
  3. The site be unavailable at the specified scheduled time.
  4. The new service will notify apps that successfully responded to the schedule endpoint to begin the migration (endpoint 2).
  5. Site will be brought live at the specified end time or once all app migrations are complete:
    a. If the migration window is over, the service will query the apps that successfully started the migration for their status (endpoint 3).
      i. If the status is ready-to-commit , the service will send a commit hook (endpoint 4).
      ii. If the status is anything else, the service will send a rollback hook (endpoint 5).
    b. or if all app migrations have completed (i.e. responded failed or ready-to-commit when polled or reported back early),
      i. If the status is ready-to-commit, the service will send a commit hook (endpoint 4).
      ii. If the status is anything else, the service will send a rollback hook (endpoint 5).
  6. Connect will then update the installed region of the apps that have been finished the migration process in the specified window.

App eligibility

The migration eligibility of an app is determined by fulfilling the following criteria:

  • The app supports data residency by supporting regionBaseUrls in the descriptor
  • The app supports the declared realm under their specified regionBaseUrls in the descriptor
  • The app supports realm migrations by supporting the dare-migration lifecycle hook in the descriptor

Supporting migrations

Descriptor changes

A new lifecycle hook will be supported in Connect for the migration process.

To support realm-to-realm migrations, you must declare the following lifecycle in the descriptor:

{
 ...,
 "lifecycle": {
  ...
  "dare-migration": "/my-migration-endpoint",
  ...
 },
 ...
}`

All endpoints will be called by appending onto the declared lifecycle hook. In the example above, when scheduling a migration, a hook would be sent /my-migration-endpoint/schedule of the app server.

Endpoints

Endpoint Description Request payload Response
Endpoint 1: POST /schedule When a product admin schedules a migration, the service will notify eligible apps via this endpoint to schedule a migration.

Use this to do any preparatory work prior to the migration.
{
 “startTime”: “2021-12-26T00:00:00.000Z”,
 “endTime”: “2021-12-27T00:00:00.000Z”,
 “location”: “EU”
}
If your app responds with a non-2xx, the app migration will be marked as a failure and will not receive further hooks (besides the rollback hook) for the same migration.
Endpoint 2: POST /start Once the site is taken offline, the service will notify eligible apps that have successfully responded to the schedule hook via this endpoint to begin the migration.
{
 “startTime”: “2021-12-26T00:00:00.000Z”,
 “endTime”: “2021-12-27T00:00:00.000Z”,
 “location”: “EU”
}
If your app responds with a non-2xx, the app migration will be marked as a failure and will not receive further hooks (besides the rollback hook) for the same migration.
Endpoint 3: GET /status This endpoint will be called by the service to fetch the status of an ongoing migration.

The statuses the service will specifically be looking for:
 - failed
 - ready-to-commit

For failed statuses, there will be a series of errorResponseCodes you may wish to report back with to provide more insight into the failure reason.

A ready-to-commit will mean that the app has finished its migration and is awaiting a /commit lifecycle upon completion.

Beyond that, you can utilize any statuses you wish; however, upon completion, any status that is not ready-to-commit will receive a rollback hook.

We will also be regularly polling the apps once the migration has started for their status. The intention is that if all apps have either failed or are ready-to-commit, then the migration can end, and the site can be brought back online.

Apps will also be able to notify the service that they’ve finished migration early.
{} Failed status - service will update the app migration’s status to failed:
{
 “status”: “failed”,
 “errorResponseCode”: “E0004”
}
App is awaiting a /commit hook. The service, once the migration ends, will send a /commit hook:
{
 “status”: “ready-to-commit”
}
Endpoint 4: POST /commit Once the site is brought back online, the service will query the status of each app that has started the migration.

The service will ask apps to commit via this endpoint if they’ve reported that they are ready-to-commit.

Specifically, this is to commit the copied data to the new region.
{} If your app responds with a non-2xx, the app migration will be marked as a failure and will receive a rollback hook.
Endpoint 5: POST /rollback Once the site is brought back online, the service will query the status of each app which has started the migration.

The service will ask apps to roll back via this endpoint if they’ve reported that they are not ready-to-commit.

Specifically, this is intended to roll back the non-destructive copy operation to the new region.
{}

Note: All outgoing requests to apps will be retried if the response is a 5xx. You must ensure your app can handle this.

Error codes

To ease problems with migrations, we’re adding a set of standardised error codes that your app can report back to us with when you’re reporting back with a non-2xx to the hooks or the status retrieval.

Error Code Reason
E0001 The migration was not scheduled.
E0002 The app does not support the target region (versioning issues).
E0003 The app does not support migrations (versioning issues).
E0004 The app has too many concurrent migrations.
E9999 Generic app failure.

Target release timeline

Currently, we are targeting Oct-Dec for an EAP release of the realm migration service. During the EAP, partners and developers can integrate with the service and start supporting realm migrations for their apps. We will provide channels to report any issues or bugs.

Please be aware, we are still firming up the timelines and will keep you updated as we get closer to the release. While we do not foresee any potential delays to the target release timeline, if we encounter any critical issues during our testing, timelines may be adjusted accordingly. We will update you if this happens.

Contact Us

If you would like to have a chat about this solution and your app use-cases, please reach out through comments, DM or email - jlo2@atlassian.com.

7 Likes

@JackyLo - what happens when a customer needs to migrate the data for a specific app without migrating their entire instance?

For example, an app adds data residency later on, and there is now a region mismatch between the app and the instance.

1 Like

Hey @ademoss!

There’s a few additional things we need to mention in this post:

  • App migrations will only be available to products which are pinned to a specific location
  • Customers cannot choose which region their app will go - it will always attempt to migrate to the same region as the pinned product
  • Customers cannot choose individual apps to migrate - which means all apps which are eligible will attempt to undergo a migration

If an installed app chooses to support data residency or new regions later - the intention of the app migration process is to also facilitate this scenario.

Say in your example the product was pinned to the EU realm and an app was installed into the global realm. If the app supports the EU region later - the customer can schedule an app migration, and as their product is already pinned to the EU, we’d only be attempting to move the eligible app’s data to the EU via the process described above.

Hope this answer your question - please let me know if it doesn’t!

Hey @JackyLo ,

Thank you for the update.

Our app is a bit special - it serves both Jira and Confluence (though different atlassian-connect.json’s, depending on the request domain). If the customer has our app installed for both Jira and Confluence, we can’t migrate just one part - it’s all or nothing.

Will it be possible for the customer to schedule a migration for both Jira and Confluence apps at the same time?

Hi, are you planning to add some capability which will allow us to test the region migration process?

4 Likes

I also would like to emphasize that we will need some way how to test the migration.

The best way would be if in our Cloud dev instances we could manually specify to which region our app should be migrated and then perform the migration. In a similar way how now we can specify the license status for our apps in dev instances.

1 Like
  1. Sounds awesome! I’d love to support data residency!
  2. Is there a (probationary) full-page documentation on residency migrations, or should we just rely on the spec above, and other bits harvested from various posts?

In other words, can you please write a documentation page with the spec above, with a banner at the top saying that this is in beta, so I can point my devs to it and ask them “Implement that spec”?

2 Likes

Hi @ErkkiLepre!

It is possible for an app migration to be scheduled for both the Jira and Confluence product for the same site at the same time. However, this is entirely dependent on the admin of the site - so there might be some co-ordination required with the admin.

We’re definitely interested in understanding the scenario here and whether concurrent app migrations support your use-case. If possible, let’s schedule a call or talk via email in greater detail!

1 Like

Hi @filip @raimonds.simanovskis thanks for your feedback!

We plan to add realm migration testing capabilities for the EAP release. We will be able to provide more information on what this will look like closer to the release.

In the meantime, if you have any specific testing use-cases you’d like to discuss in more detail, please let myself or @JackyLo know and we can schedule a call. Thanks!

Hi @aragot!

Great to hear you’d like to support data residency!

Just a FYI but data residency realm-pinning for Connect apps is currently available, however, realm migrations for Connect apps is currently not available.

Unfortunately this means for realm migrations, there’s no official or probationary documentation yet since it’s not officially released or in beta.

This post was to provide a heads-up on what we’ve been working on delivering and we’ll have official documentation available upon the EAP release.

2 Likes

Hi @JackyLo , I have send you an email.