Data Residency for Marketplace Apps Status Update

Hi @PurnaChandraBoyapati,

To fully support the various use cases of data residency both the regionBaseUrls and the dare-migration field have to be added to the Connect descriptor.

This is how they will be used:

  • the URLs defined as regionBaseUrls control the location/realm when an app is installed. If a customer moved a product to a specific location, when an app is installed, the corresponding location from the Connect descriptor will be selected. Even if the product has not been been moved to a specific location, the regionBaseUrls is used at app installation time if the app supports the location where the product is provisioned. More information can be found on the (Data residency developer documentation).
  • the lifecycle dare-migration field are used when a customer requests a migration for the app data. This happens when the app was installed in the Global location because either the product was not migrated yet or because the app didn’t support data residency. In cases like these, the customers will have to request the app data to be moved. The Advance Notice: Connect Data Residency Customer Rollout topic shows the UX for customers and the data residency migrations contains the details of the required hooks.

To recap, while there is no validation on the descriptor and only providing one or the other field is technically possible, both are required to support all possible scenario.

Hope this helps,
Caterina

3 Likes

Hi @ccurti , I hope you’re well.

I’m wondering if we could have an update about the roadmap shared above? I’m particularly interested in this item: Realm pinning persistence for app re-installs. That’s currently a blocker for us (Zephyr Scale) to release data residency.

Thank you,
Vitor

Hey @VitorPelizza,

Apologies for the delayed response, this slipped by. We recently announced that Data Residency was available for Admins under a beta label. Subsequent to this release, we have been working on ways better communicate the status of Apps, particularly where they do not store data outside of Atlassian.

While we certainly want to explore the persistence of realms on app reinstallation, it is not something which we have planned for the short to medium term. It would be great to learn more about why you see this as a blocker and if there are any other considerations we should make as we explore this functionality. Please feel free to book some time in my calendar to connect and discuss this (or general data residency in Connect) further.

Thanks,
Sean

Hi Sean,

Thank you for your answer. Unfortunately, this is not the answer I was hoping for. Including on this page but also seen in other materials, including during conversations at AppWeek@Berlin, this item is on the roadmap. I understand it’s never been on the short-term roadmap, but this conversation have been going on for 6 months or so now, so I was hoping that this would be somewhere near to get prioritised.

This behaviour is critical for us to release data residency for Zephyr Scale. Without this bug fix (it is a bug in my opinion), our customers might get pointed to an empty installation of Zephyr Scale if they reinstall their app after we release data residency. If that scenario happens, we won’t be able to move their data over, particularly not without intervention.

Unfortunately, I can’t find a suitable time in your agenda as I’m in the UK. Would you mind to share some European-friendly times?

Many thanks,
Vitor Pelizza

2 Likes

What do we do with dare-migration if all app data is only stored in Atlassian storage that already supports data-residency and location migration?

Hi @daviddrawio,

we shared this in the Advance Notice: Connect Data Residency Customer Rollout topic

  • What if I store data in Jira and Confluence?
    • If you store data exclusively within Jira and Confluence, the data will inherit data residency controls already available for these products, therefore you do not need to support Connect Data Residency realm pinning and migrations.

However, as a few commented there, with the current UI the app will appear as Not Eligible to be pinned or migrated. The following is presented to the users when browsing to that tab and the data stored exclusively within the Atlassian product is the first item in the bullet list:

This is something that the team is considering improving (as mentioned here).

Caterina

2 Likes

This page https://developer.atlassian.com/cloud/jira/platform/data-residency/

Says

EU: In-scope data is hosted within the Dublin AWS regions

So if the data is in Germany they don’t count as EU? #confused

From my understanding, this is just where Atlassian stores their EU data, not which AWS Regions are allowed for EU.

2 Likes

Hi @SeanBourke ,
I have a question about the app descriptors for Connect (i.e. atlassian-connect.json).

Which descriptor (i.e. the descriptor from which region) is used for the marketplace version and update checks?

What happens if the regional app descriptors differ?

Hey @marc,

You should only need to define one single app descriptor for your Connect app. This descriptor should include the regionBaseUrls for each of the regions which are supported, with a different baseUrl which defines where Connect will route any app traffic for customers who have installed the app within that region.

This would be the same descriptor which is used today, as there’s no need to define a descriptor for each of your regions - your descriptor should include information about which regions are supported and where traffic should be routed for these.

Thank you for the explanation. I was under the impression all regional app instances needed to serve atlassian-connect.json.

Hi,

We were trying to implement data residency in a connect app and stumbled upon a situation that upon finding this thread we realized it had already been identified and has been described above as “realm pinning persistence for app re-installs”.
When an app is re-installed we need Jira to keep calling the app through the same URL as before it was uninstalled, even if the app wasn’t pinned.
And even if we define a “global” region in the app descriptor, Jira will pick another region if the app isn’t pinned into a specific region and is set to the Global region.

Are there any plans to change this?

Best regards,
Sérgio Vieira

4 Likes

Hello,

Can you clarify (and add it to documentation) on which base url the webhooks are called (global/target realm/source realm) and whether all webhooks are called on the same base url during specific migration?

Hey @sergio.vieira,

Thanks for sharing your concerns and understand the constraint. We have previously shared details about our thinking around this. Our primary concern is the diversity and complexity of implementations of realm pinning and realm migrations to date - we need to consider a solution which is suitable for developers who may rely upon this logic for their use cases as well.

While I don’t have any timing about when we’re likely to deliver this, I can commit to us utilising the RFC process to ensure that we assess the feasibility of the solution with the developer community and source their feedback to ensure we’re heading in the right direction.

1 Like

Hey @lexek-92,

Thank you for your feedback, appreciate there is an opportunity to make this clearer within our documentation.

All hooks for the Data Residency lifecycle events will be emitted to the original realm (that is, the realm which the customer requesting the migration is currently located), including the /commit hook. If the migration is committed, then once the host product is made available again, all subsequent traffic for that customer will be directed to the target realm (the one which has been migrated to). If the migration is rolled-back, then all subsequent traffic once the host product is made available again will be directed to the original realm.

1 Like

Hi Sean,

Is there a retry policy for /rollback webhooks? In broader terms, what happens if rollback returns non-2xx response?

Also, what is expected behaviour if /status returns non-2xx response and/or response in non-compliant format? Will it also cause a rollback?

Is there maybe a possibility to consider retry for intermittent 502/503 responses?

Hi,

  1. Is there any way to test Data Residency on a dev instance? If i upload an app by providing the atlassian-connect.json file will it be forwarded to one of the regions defined in the regionBaseUrls (defined in atlassian-connect.json)? Is there any way to debug how Jira forwards the install request?
  2. if the app doesn’t yet support realm migration will it still be eligible for Data Residency?

Hey @JulianVasaK15t,

Thanks for the questions, I’ve provided some answer in-line below. Hopefully these help.

Data residency on a dev instance should work similarly to any other instances. If you have pinned the product to a region which is supported in you app’s descriptor, you should see traffic for a new installation directed to that regionBaseUrl. If you haven’t pinned the product, the region will be selected based upon the product provisioned region (shard) if it is supported in your descriptor. If you do not support the product region or the provisioned region, the requests for a new installation should be directed to your baseUrl.

Unfortunately the determination of this routing is performed at installation time and cannot be debugged externally; however you can see the current location of an app following its installation in the data residency app details screen.

An app which supports realm pinning but not realm migration may appear to customers as either Pinned (when it is already installed in the same region as the customer’s product) or Not Eligible (when it is hosted in a different region to the customer’s product, or does not support region of the customer’s product). Once realm migration is supported, if the customer’s product region is supported, the app will appear in Eligible and be available to have a move requested.

1 Like

Sorry @lexek-92, I missed this query.

/rollback currently implements retries to notify your app, reattempting twice after the initial failed attempt.

We’re currently improving the retry behaviour to other polling hooks to ensure that transient errors are less likely to result in a complete migration failure. Where an error code is present we won’t attempt any retries; however all other endpoints which could result in failure will retry to ensure best-efforts to not fail the migration.

Hi @lexek-92,

An update on this - we’ve released some improvements to the retry mechanisms for the data residency migration hooks. The related documentation has been updated to reflect this improved behaviour.