RFC-22: Realm Persistence in Connect Data Residency

[SeanBourke]

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building shared understanding of a topic. It expresses a technical solution, but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Summary of Project:

Realm Persistence will support partners who have already, or plan to adopt Data Residency for their Connect App, by helping to ensure that subsequent uninstallation/reinstallation events will persist to the same realm.

• Publish: 2023-08-24T14:00:00Z
• Discuss: 2023-09-07T14:00:00Z
• Resolve: 2023-09-14T14:00:00Z

Problem

Following the availability of Data Residency for Connect we’ve heard partner concerns about potential scenarios where customer’s app installations may target a different realm following reinstallation. There are two key circumstances under which this can occur:

Scenario 1:

1. a customer already has an existing app installed (targeting the app’s baseUrl)
2. that app adopts data residency
3. the customer uninstalls the app (notifying the baseUrl)
4. the customer subsequently reinstalls the app (pointing at a different regionBaseUrl)

Scenario 2:

1. an app adopts data residency
2. a customer installs that app into their current product’s region (pointing at the same regionBaseUrl)
3. the customer uninstalls the app (notifying the regionBaseUrl)
4. the customer migrates their product to a different region
5. the customer subsequently reinstalls the app (pointing at a different regionBaseUrl)

When this occurs, the app is reinstalled in (and traffic is routed to) a different regionBaseUrl , but the app data may still be in the app region where the app was originally uninstalled.

For partners who have accommodated for this behaviour, this is not an issue. However, if this situation is not accounted for, customers may experience a temporary loss of data/configuration if/until the partner manually migrates their data.

Who are we solving for?

For Marketplace Partners: Realm persistence helps partners save time and avoid negative customer experience by helping to ensure that customer apps is unlikely to unexpectedly move between realms. With realm persistence, app data will only move when reinstalled outside of a defined duration or when otherwise requested to move by the customer through the realm migration service.

For customers: Realm persistence helps in reducing the likelihood of data loss when a customer reinstalls an app, by ensuring that it persists to here their data was previously stored. This is necessary for customers when using an app which otherwise does not have mechanisms or controls in place to detect and automatically migrate data to a different realm following reinstallation.

Proposed Solution

Definition in Descriptor

We are proposing that Realm Persistence can be defined within your Connect descriptor in a new declarative section, as below. By default, this behaviour is false to persist the behaviour of existing data residency implementations.

Initially, this declaration of realmPersistence is proposed to be optional, however to ensure explicit definition and to ensure that partners who adopt data residency are aware of the considerations regarding realm movement, it may become mandatory after some time.

{
  // Existing field for base URL
  "baseUrl": "https://your.sample.app.com"
  ...
  ...
  // Existing field for data residency
  // https://developer.atlassian.com/cloud/confluence/data-residency/#enable-data-residency
  "regionBaseUrls": {
    "US": "https://us.your.sample.app.com",
    "EU": "https://eu.your.sample.app.com",
    "AU": "https://au.your.sample.app.com",
    "DE": "https://de.your.sample.app.com"
  }
  ...
  // New field data residency options
  "dataResidency": {
    "realmPersistence" : true, //optional
    "realmPersistenceDays": 30 //optional
  }
  ...
}

Duration of Persistence

The duration of realm persistence can also be defined within the Connect descriptor the the realmPersistenceDays property. This is bound to a minimum of 1 day to a maximum of 30 days and defaults to 7 days when otherwise not defined in the descriptor.

Where an invalid response is provided, the default of 7 days will be utilised if realmPersistence is true.

Asks

While we would appreciate any reactions you have to this RFC (even if it’s simply giving it a supportive “Agree, no serious flaws”), we’re especially interested in learning more about:

• Does the availability of Realm Persistence increase the likelihood of you supporting Realm Pinning and Realm Migration within your app?
• Do you have any blocking use case that will stop you from using the proposed solution to support realm persistence with data residency within Connect?
• Does the proposed adoption method and default behaviour appear reasonable? Are there concerns around the proposed requirement of this field in the future where data residency is already utilised by an app?
• Partners utilising Realm Persistence may need to provide awareness to customers about their app’s behaviour (i.e. updates to their data processing policies). Is this reasonable, or a potential blocker to adoption?
• How do you feel about the problem we are trying to solve and do you have any suggestions for us on this RFC?

[danielwester]

This looks great to me. I do have a question (or two).

1. How will this be communicated to the customers? Will it be when they uninstall it that the vendor will keep the data for realmPersistenceDays days? (Which might not be correct)…
2. For us, we’re storing the data as long as we have a business relationship with the customer (ie they’ve got a valid subscription). So ideally if they’ve got an active subscription - you persist their realm for the entire period.

Thanks,

/Daniel

[markrekveld]

I like the approach, no major red flags, but think the realmPersistenceDays with a max of 30 may a bit on short side.

In my app I keep data for 60 days after a customer uninstalls the app, so the scenarios for my app still apply if the reinstall dans is done between day 31 and 60.
It would be great if the data persistence duration can be in-sync with the realm persistence duration.
Even if this means that the data persistence duration cannot be greater then 30 days in the case the realm duration cannot be increased.

On the customer side however, how is he/she updated on the persistence duration? Will this be something that is displayed within there Jira/Confluence so they see like a countdown till realm persistence is cleared?

Cheers,
Mark

[satvik]

I agree with @danielwester, this solution should cover their entire subscription.

However, I’m surprised this was ever something that partners had to accommodate, and that this proposal only reduces the likelihood of needing to do a manual data migration.

In my opinion, Atlassian should be persistently storing which realm apps were previously installed in. Customers already have to opt into migrating apps into their chosen realm separately from their site, and both the current state or realm persistence as proposed have situations where they don’t always have that control.

I can think of two ways which keep the customers in control.

1. When re-installing an app, customers are prompted to choose their current region or the previously installed region.
   a. This gives the customer more control and information about what’s happening.
   b. This provides a place to let the customer know that if they install to the new region, their old data could disappear.
2. Apps always reinstall into the realm they were uninstalled from, and customers get a notification that they have to go and migrate the app into their current realm if it’s out of sync.

[VitorPelizza]

This indeed looks great, thanks a lot for this RFC!

Does the availability of Realm Persistence increase the likelihood of you supporting Realm Pinning and Realm Migration within your app?
Yes, absolutely. We’ve been concerned about rolling out Data Residency without this proposed solution in place. From our perspective, the current behaviour looks like a bug because customers would be moved from one realm to another without triggering any migration.

Do you have any blocking use case that will stop you from using the proposed solution to support realm persistence with data residency within Connect?
No, we will use it straight way.

Does the proposed adoption method and default behaviour appear reasonable? Are there concerns around the proposed requirement of this field in the future where data residency is already utilised by an app?
From our perspective, the proposed solution makes the data residency mechanism more robust and predictable. It fixed an unexpected behaviour of moving customers to a different realm without triggering any migrations. So, no concerns here, on the contrary!

Partners utilising Realm Persistence may need to provide awareness to customers about their app’s behaviour (i.e. updates to their data processing policies). Is this reasonable, or a potential blocker to adoption?
It is not. In fact, the use case that requires the proposed solution is a bit of a corner case, so it is not going to have an impact on the majority of the customers. Also, the proposed solution doesn’t block customers from migrating the app to another region afterwards, so I don’t believe that there is an impact in terms of data processing policies.

How do you feel about the problem we are trying to solve and do you have any suggestions for us on this RFC?
We are very excited to see this being rolled out. I think that the only suggestion is to make the proposed behaviour the standard behaviour and use the flags in the descriptor to opt-out and instead make apps change automatically their realms if they fall into the described use case. This is a similar note to what @satvik mentioned.

Thanks a lot, we are looking forward to other’s comments and see this moving forward

[SeanBourke]

Thanks everyone for the feedback thus far, really appreciate your consideration and thoughts. Some of the question we’ve received have been similar, so I’ll answer them together in-line.

@danielwester

How will this be communicated to the customers? Will it be when they uninstall it that the vendor will keep the data for realmPersistenceDays days? (Which might not be correct)…

@markrekveld

On the customer side however, how is he/she updated on the persistence duration? Will this be something that is displayed within there Jira/Confluence so they see like a countdown till realm persistence is cleared?

As noted in our questions, partners utilising Realm Persistence may need to provide awareness to customers about their app’s behaviour (i.e. updates to their data processing policies). An app’s data security and privacy link is provided when a Connect app is uninstalled (refer to the sample below).

As @danielwester pointed out, alignment to a realm is very different to an app’s data retention policies. We’re conscious that the display of this within the uninstallation dialogue may be easily misconstrued and thus this is something we’d likely explore further as an enhancement only if we saw considerable adoption of this capability.

@danielwester

For us, we’re storing the data as long as we have a business relationship with the customer (ie they’ve got a valid subscription). So ideally if they’ve got an active subscription - you persist their realm for the entire period.

@markrekveld

In my app I keep data for 60 days after a customer uninstalls the app, so the scenarios for my app still apply if the reinstall dans is done between day 31 and 60.
It would be great if the data persistence duration can be in-sync with the realm persistence duration.
Even if this means that the data persistence duration cannot be greater then 30 days in the case the realm duration cannot be increased.

This solution will assist in more common scenarios, such as ones where customers may face an issue with an app and then uninstall/reinstall it in an attempt to remedy this (or be guided to do so by the partner’s support team).

While we appreciate that there may be customers who uninstall/reinstall beyond the 30 day period, the predominant proportion of reinstallations occur within hours or a few days. We need to consider what upper limits make sense - 30 days covers an extremely high % of reinstallation scenarios.

[SeanBourke]

@satvik

Apps always reinstall into the realm they were uninstalled from, and customers get a notification that they have to go and migrate the app into their current realm if it’s out of sync.

@VitorPelizza

I think that the only suggestion is to make the proposed behaviour the standard behaviour and use the flags in the descriptor to opt-out and instead make apps change automatically their realms if they fall into the described use case. This is a similar note to what @satvik mentioned.

We’re conscious that the migration of an app to a different realm subsequent results in product and app downtime (up to 24 hours) for customers. Not all apps require realm persistence and some apps which have implemented realm pinning today may have already introduced measures to account for this scenario, ensuring that customers will always be aligned to their product realms upon any reinstallation. If a customer’s app installation can be aligned to their product realm, that reduces the likelihood of requiring a migration (and subsequent downtime); however we appreciate that this is not achievable in every circumstance.

By making this behaviour opt-out rather than opt-in, we would need to consider the change as a behaviour to our existing Realm Pinning implementation and would require a sufficient deprecation window to support apps to opt-out of this behaviour.

Regardless, we have noted that the field may become mandatory in the longer term. We want to ensure that partners who implement data residency are aware of the circumstances of reinstallations and therefore make an informed decision about whether to / not to implement realm persistence.

[satvik]

Hi Sean, I really appreciate your response. I’m well aware that migrating an app requires a downtime for customers and that avoiding that would be ideal, and the proposed solution wants to minimises the number of customers who have to migrate apps because a re-installation went to the original realm. I also understand wanting to avoid making breaking changes to the Realm Pinning implementation as it’s also a big hassle and requires deprecation windows.

At the end of the day, we all want good experiences for our shared customers, and with what you’re saying, 30 days covers a very high percentage of re-installations, and that this is not a very common scenario to begin with.

My first response to this is, for vendors, it’s not just the percentage that matters, but that if it does happen, any impact and required effort is on us as the vendor, but I would say it’s the Connect framework/the underlying platform which has failed the customer, not us. I would even say that any partners that have already accommodated for this behaviour are working around the edge cases of the provided platform, and that they shouldn’t have had to put that effort in on top of supporting Realm Migration as documented.

Frankly, the main problem I have with this RFC (while an amazing step in the right direction) is that to me there isn’t really any meaningful difference between having to, probably manually, handle this for some number of customers vs a fewer non-zero number. I think that the solution should either (as I originally said) give customers the agency to make the decision on installation, which would give us a point at which vendors could provide guidance/documentation or provide platform features to let vendors know they may need to migrate data across realms from a previous installation when a customer installs their app.

I really do appreciate that the proposed solution doesn’t want to provide more work for vendors as part of implementing Realm Persistence, but unless it can guarantee that we’ll never have to deal with these edge cases, it shifts it to one off support work which is also likely to affect our relationship with any affected customers.

The first idea which comes to mind is that if the data around which realm an app was uninstalled from was stored persistently it could become part of the /install webhook payload to flag to vendors that the app was previously installed in a different realm, and atlassian-connect-express would have to be updated to expose this. This would be far easier for a vendor to action vs having to architect their infrastructure in a way which makes it easy to find data from a previous install in another realm, or somehow tacking it on.

Thanks,
Satvik
Platform Engineer @ Easy Agile

[danielwester]

I’m going to join @satvik in his concerns.

The fact that Atlassian isn’t going to persist the realm location (and I don’t understand why) for the duration of the lifetime of the subscription means that this feature becomes worthless for the majority/serious partners. We will still have to store the realm location on our side because of that and rely on that instead of Atlassian’s storage.

In fact because we will have to do this - this feature becomes dangerous for the partners. Atlassian might say that the data is being stored in EU to the customer but the partner might store it in Singapore because that’s where they were told to store it initially. This split-brain issue then opens up liability issues for the partners.

So either implement it properly where Atlassian is the Trusted Source the entire time (we’ll need audit trail as well btw - but that’s probably another issue) or allow partners to opt-out.

[markrekveld]

@SeanBourke

extremely high % of reinstallation scenarios

Just covering X percentage of the scenarios is not enough from a partner standpoint, as @satvik also indicated. This still leave the partner with either having to implement a manual or automated process to cover the “edge” cases.

A manual process will be error prone since us humans tend to make mistakes when it comes to copy & paste.
An automated process will always need to be executed, because on the partner side we either don’t know the history of realm migrations, or need to execute the process to verify if data needs to be copied/processed if er store the realm.

Either way, this leaves customers with a less then ideal experience.

The ‘to be implemented’ solution should cover all cases, and looking at this thread I would like to suggest that the realmPersistence in the description becomes a simple opt-in/opt-out flag field. If the partner opts-in to, then store it indefinitely.

[jens]

Thanks for bringing this up and also for the suggested solution which is certainly a step in the right direction!

I also agree with the concerns related to the persistence duration. The last realm should ideally be stored indefinitely or at least as long as there’s an active subscription. What’s the purpose of limiting it (besides saving a tiny amount of storage space)?

Also why is this opt-in in the first place? I wonder if there is a valid reason at all for an app to prefer the current situation. Unless there are such reasons the current approach only increases complexity for app vendors without any benefit for the customers. Enabling indefinite persistence by default instead would automatically fix this class of problems for all apps. If absolutely required an opt-out could also be implemented instead.

[scott.dudley]

@SeanBourke

While we appreciate that there may be customers who uninstall/reinstall beyond the 30 day period, the predominant proportion of reinstallations occur within hours or a few days. We need to consider what upper limits make sense - 30 days covers an extremely high % of reinstallation scenarios.

I too appreciate the thought that went into this RFC.

To reframe the question here of “considering what upper limits make sense”: why does Atlassian need to impose an upper limit at all? Why not infinity? What problem is this intended to solve? (Database rows are cheap.)

I think Satvik’s comment is right on the money: either allow customers to choose a region upon installation and show them the previous region (probably a lot of work on your part given the multiple products involved), or else always install to the previous region and notify the customer that this is happening and that it can be migrated. Now that apps can be moved from one region to another via the admin control panel, the latter seems like it could be the sweet spot between reducing Atlassian work but not FTCing.

[scott.dudley]

I also think the default/legacy behavior (meaning nothing declared in the connect.json) is unhelpful and I am not sure of why anyone would want to depend on it, now that automated realm migrations are available. You have my vote for announcing a deprecation of the legacy behavior at the same time as you launch this.

[lexek-92]

Hi @SeanBourke,

It’s a bit unclear what is the expected behaviour when site is re-created (eg site import, sandbox re-creation and etc) when new “client key” is generated for the same site, whether realm still will be persisted or it will be considered as completely new installation outside of realm persistence loop.

[AurelioPrezNoriega]

Hi!

These are the answers we have from our point of view:

Does the availability of Realm Persistence increase the likelihood of you supporting Realm Pinning and Realm Migration within your app?
We are committed to addressing data residency regardless. However, this enhancement will enable us to deliver even greater satisfaction to our customers.

Do you have any blocking use case that will stop you from using the proposed solution to support realm persistence with data residency within Connect?
We prefer not to set a maximum limit for the ‘realmPersistenceDays’ property, as it could potentially lead to data loss in certain scenarios.

Does the proposed adoption method and default behaviour appear reasonable? Are there concerns around the proposed requirement of this field in the future where data residency is already utilised by an app?
This seems reasonable, and we do not have any concerns.

Partners utilising Realm Persistence may need to provide awareness to customers about their app’s behaviour (i.e. updates to their data processing policies). Is this reasonable, or a potential blocker to adoption?
Transparency is essential when it comes to informing customers about how their data is stored and processed. This should not be an obstacle to adoption if we communicate the process effectively.

How do you feel about the problem we are trying to solve and do you have any suggestions for us on this RFC?
We are highly satisfied with the ongoing work on this solution, as it will undoubtedly boost end-user satisfaction. We currently have no further suggestions.

Thanks!!

[SeanBourke]

@lexek-92

It’s a bit unclear what is the expected behaviour when site is re-created (eg site import, sandbox re-creation and etc) when new “client key” is generated for the same site, whether realm still will be persisted or it will be considered as completely new installation outside of realm persistence loop.

Hey @lexek-92, great question. Any actions which generate a new clientKey are considered fresh installations within Connect (you can refer to a write-up about this here). Given these are considered a fresh installation, their realm would map as if it had no existing installation. Is this the behaviour which you expected?

[SeanBourke]

Thank you everyone for the additional comments, thoughts and feedback over the past few days.

It sounds like there’s an overwhelming desire to support a longer duration (or no limits) to ensure that the utility of this is broader reaching and eliminates the need to implement a workaround (either manual or automated). We’ll take this feedback and explore it more thoroughly to understand the implications and what is feasible.

One consideration to be made here is that realm persistence is predominantly associated with realm pinning (ensuring data points to the correct realm), but a distinct upper limit also defers to realm alignment upon reinstallation. I would be keen to understand:

1. If there was a need to implement an upper limit for this, what would you likely configure for your apps?
2. Would a coupling of supporting realm migrations alongside realm persistence impact your desire or ability to adopt data residency?

With regards to this being a default behaviour, we are conscious that other partners may have already implemented behaviours and measures to accomodate the lack of realm persistence. Regardless, adoption of this capability is a great indicator of its importance to the community and if we see it being a predominant default in apps adopting data residency, that suggests that an associated change in default behaviour / deprecation may be warranted.

[markrekveld]

1. If there was a need to implement an upper limit for this, what would you likely configure for your apps?

It’s all about the data of the customer, so for me this would be linked to the data retention period that kicks in after a customer uninstalls the app. In my case this would be 60 days.
After the data of the customer is deleted, I don’t really care were there next install will lead the data to. But as long as the app holds data of a customer any lifecycle event for the app should always go to that realm.

2. Would a coupling of supporting realm migrations alongside realm persistence impact your desire or ability to adopt data residency?

Not having proper realm migrations is blocking me from adopting data residency at the moment.
This realm persistence improvement is the last bit for me to adopt data residency as its completes the migration feature set for the data residency I think.

[satvik]

Just another thing that’s popped up while we’re trying to test our Data Residency implementation.

The existing behaviour is super helpful for testing the Connect integration with Atlassian while we haven’t yet built support for migrations, so as Marketplace Partners, having an escape hatch to the current behaviour on a specific Jira instance would be super helpful.

[SeanBourke]

Hey @satvik,

Thanks for the feedback and very important note. We’ve considered that this may require some pathway to expedite moves. Our current exploration of this suggests that the appropriate pathway would be a request via a developer support request.