Hi, I’m Jacky, an engineer at Atlassian. Our team is working on data residency migrations for Connect apps. Back in 2020, we sought out some feedback on realm-to-realm migrations for Connect apps in the Atlassian Developer Community. We’re now looking to provide you with the following:
- some assumptions we’re making,
- more details on the migration endpoints we’ve been working on,
- and an update on the app migration process.
Note, this is only a preview of the functionality we’re building, and is not a confirmation of its final implementation. We’ll provide official documentation of the functionality once it’s fully released.
Assumptions
Below are the assumptions we’re making:
- App migrations will be done while the site is down.
- No apps will be uninstalled as a result of the site migration process.
- App migrations will be a non-destructive copy operation.
- App migrations will not block site migrations.
- App migrations will be performed at a site-level.
- The eligibility of an app to be migrated will be decided at the scheduling time.
Revisiting the app migration process
In our previous post in the Atlassian Developer Community, we briefly described the realm-to-realm migration process that we envisioned at the time. With the above assumptions, the process has more or less stayed the same, with the exception of the following minor changes:
- A new service will be introduced to orchestrate the migration process for apps. The service will continue to use JWT auth — signed with the sharedSecret you’ve obtained in the installed payload.
- A rollback hook will be introduced to notify apps that the migration is not to be processed and the original copy of the data will be used.
- The status hook serves a slightly different role. More details about this below.
- Custom error response codes have been introduced to help diagnose failures.
With these updates, the process will now be as follows:
- The product admin will schedule a migration of eligible apps.
- The new service will notify eligible apps via the schedule endpoint (endpoint 1).
- The site be unavailable at the specified scheduled time.
- The new service will notify apps that successfully responded to the schedule endpoint to begin the migration (endpoint 2).
- Site will be brought live at the specified end time or once all app migrations are complete:
a. If the migration window is over, the service will query the apps that successfully started the migration for their status (endpoint 3).
i. If the status isready-to-commit, the service will send a commit hook (endpoint 4).
ii. If the status is anything else, the service will send a rollback hook (endpoint 5).
b. or if all app migrations have completed (i.e. respondedfailedorready-to-commitwhen polled or reported back early),
i. If the status isready-to-commit, the service will send a commit hook (endpoint 4).
ii. If the status is anything else, the service will send a rollback hook (endpoint 5). - Connect will then update the installed region of the apps that have been finished the migration process in the specified window.
App eligibility
The migration eligibility of an app is determined by fulfilling the following criteria:
- The app supports data residency by supporting
regionBaseUrlsin the descriptor - The app supports the declared realm under their specified
regionBaseUrlsin the descriptor - The app supports realm migrations by supporting the
dare-migrationlifecycle hook in the descriptor
Supporting migrations
Descriptor changes
A new lifecycle hook will be supported in Connect for the migration process.
To support realm-to-realm migrations, you must declare the following lifecycle in the descriptor:
{
...,
"lifecycle": {
...
"dare-migration": "/my-migration-endpoint",
...
},
...
}`
All endpoints will be called by appending onto the declared lifecycle hook. In the example above, when scheduling a migration, a hook would be sent /my-migration-endpoint/schedule of the app server.
Endpoints
| Endpoint | Description | Request payload | Response |
|---|---|---|---|
Endpoint 1: POST /schedule |
When a product admin schedules a migration, the service will notify eligible apps via this endpoint to schedule a migration. Use this to do any preparatory work prior to the migration. |
|
If your app responds with a non-2xx, the app migration will be marked as a failure and will not receive further hooks (besides the rollback hook) for the same migration. |
Endpoint 2: POST /start |
Once the site is taken offline, the service will notify eligible apps that have successfully responded to the schedule hook via this endpoint to begin the migration. | |
If your app responds with a non-2xx, the app migration will be marked as a failure and will not receive further hooks (besides the rollback hook) for the same migration. |
Endpoint 3: GET /status |
This endpoint will be called by the service to fetch the status of an ongoing migration. The statuses the service will specifically be looking for: - failed- ready-to-commitFor failed statuses, there will be a series of errorResponseCodes you may wish to report back with to provide more insight into the failure reason. A ready-to-commit will mean that the app has finished its migration and is awaiting a /commit lifecycle upon completion.Beyond that, you can utilize any statuses you wish; however, upon completion, any status that is not ready-to-commit will receive a rollback hook. We will also be regularly polling the apps once the migration has started for their status. The intention is that if all apps have either failed or are ready-to-commit, then the migration can end, and the site can be brought back online.Apps will also be able to notify the service that they’ve finished migration early. |
{} |
Failed status - service will update the app migration’s status to failed:/commit hook. The service, once the migration ends, will send a /commit hook: |
Endpoint 4: POST /commit |
Once the site is brought back online, the service will query the status of each app that has started the migration. The service will ask apps to commit via this endpoint if they’ve reported that they are ready-to-commit.Specifically, this is to commit the copied data to the new region. |
{} |
If your app responds with a non-2xx, the app migration will be marked as a failure and will receive a rollback hook. |
Endpoint 5: POST /rollback |
Once the site is brought back online, the service will query the status of each app which has started the migration. The service will ask apps to roll back via this endpoint if they’ve reported that they are not ready-to-commit.Specifically, this is intended to roll back the non-destructive copy operation to the new region. |
{} |
Note: All outgoing requests to apps will be retried if the response is a 5xx. You must ensure your app can handle this.
Error codes
To ease problems with migrations, we’re adding a set of standardised error codes that your app can report back to us with when you’re reporting back with a non-2xx to the hooks or the status retrieval.
| Error Code | Reason |
|---|---|
E0001 |
The migration was not scheduled. |
E0002 |
The app does not support the target region (versioning issues). |
E0003 |
The app does not support migrations (versioning issues). |
E0004 |
The app has too many concurrent migrations. |
E9999 |
Generic app failure. |
Target release timeline
Currently, we are targeting Oct-Dec for an EAP release of the realm migration service. During the EAP, partners and developers can integrate with the service and start supporting realm migrations for their apps. We will provide channels to report any issues or bugs.
Please be aware, we are still firming up the timelines and will keep you updated as we get closer to the release. While we do not foresee any potential delays to the target release timeline, if we encounter any critical issues during our testing, timelines may be adjusted accordingly. We will update you if this happens.
Contact Us
If you would like to have a chat about this solution and your app use-cases, please reach out through comments, DM or email - jlo2@atlassian.com.