When should I be deleting a tenant's data?

#1

So building out an app and it’s time for the “oh crap - I’ve got data that I need to delete when the tenant is no longer a customer- when should I delete it?” discussion. Some of this is a recap from discussions I’ve had with other vendors and what I’ve done in the past - but I would love to get Atlassian to speak up (and maybe prompt Atlassian to have some public discussions about it).

The general approaches

/uninstall directly

The easiest approach is to delete when the customer uninstalls the app. But what if they’re just temporarily removing the app? Or worse - the uninstall hooks gets accidentally triggered…

/uninstall with a delay

To combat that - wait 30 days or something after the uninstall. Then delete. But… not all instances trigger a /uninstall - if Atlassian deletes the instance - we don’t get the /uninstall call.

poll for access

This is the most official suggestion I’ve heard. Basically poll each day and then after X days of not having access - delete the data.

look at the license data

Have the app connect up to the marketplace (using the non app token passwords) and download the license report and then match up the SENs. Then at expiration - delete the data. However the license data in the marketplace doesn’t always match up with the license in the app instances (jira has a couple of days longer).

Also connecting an app up with the marketplace like that scares the crap out of me from a security perspective.

Don’t F the customer

Polling and uninstall hooks all seem to also to potentially be a way to “f* the customer”. As a customer I might have temporarily uninstalled the app for whatever reason but I’m still paying for the license. I would at that point expect my data to stay (since I’m paying for it).

PS. Let’s not forget the data that’s stored in the instance

I would be amiss if I didn’t mention all of those content properties that I might have set in the instance - after I’m uninstalled or lose access - I can’t delete those. :frowning:

So what to do?

I really don’t want to have any data longer than necessary but at the same time I can’t come up with a good way of detecting when to delete things.

Help?

16 Likes
How to delete stored User Properties, Issue Properties etc. on Connect Addon Uninstall?
#2

Tagging folks at Atlassian that might be interested in the topic (or know somebody that might). @akassab @rwhitbeck @nmansilla @mpaisley

#3

Thank you @danielwester for raising this issue. I have struggled with that before: ACE and uninstall lifecycle event .

Clear guidance from Atlassian would help a lot.

#4

@danielwester, thank you for sharing!

We use the " /uninstall with a delay" option.

However, you just made me sad with the fact that we do not get it “if Atlassian deletes the instance”. We didn’t realize that before. I feel that Atlassian should fire the “/uninstall” or some other webhook in this case.

Cheers,
Jack

#5

It gets even worse - if an instance is restored - you’ll get a new installation with a new clientKey (but same SEN and license as before). Which means that you have zombie data sitting in your database until you manually clean it up.

#6

To try and mop up all the edge cases we also track the last time the install hook was fired and manually publish a new descriptor every quarter to see what tenants are absent

2 Likes
#7

Same here. We also have a hard time cleaning up data of instances for which we didn’t receive uninstall webhooks.

We try to clean up by checking the licensing status, but obviously this only works for licensing-enabled apps.

Even detecting if an instance that we have a tenant record for still exists is not trivial at all because there aren’t any defined HTTP response codes when polling that instance’s base url (the instance could be active, gone, sleeping, temporarily not available, …).

Some ideas:

  • It would be really great if Atlassian could provide a central (not on the instance) endpoint where apps could retrieve the status of an instance for example by providing the instance ID.
  • Another nice-to-have could be a /request-data-removal webhook that can be triggered by instance admins in the UPM for each app. This would enable them to manually delete their data before they uninstall an app.
5 Likes
#8

Our app ProForma stores all data in Jira entity properties, so when a customer uninstalls we cannot access their data any more.

That solves the problem for us because we don’t hold anything once a customer has left, but it could be a problem for them. If they want to get rid of their ProForma data after they uninstall then they’re too late — we can’t access their Jira instance any more so we can’t delete it. Jira doesn’t know what things our app wrote so it can’t delete it either.

Perhaps one way to solve it would be for Atlassian to reinstate our app’s access to Jira APIs when someone requests delete. Perhaps better, though, would be if data our apps store in Jira is marked as being owned by the app. That way Jira could provide a button for administrators to delete it without our app having to be involved.

100 app property limit should be lifted
#9

@daniel

Just chiming in to understand this better. Am no expert in this topic, but I would say I have vested interest in it :slight_smile:

How will a customer continue to pay for license after uninstall? I assumed Atlassian will only invoice them till the day an app is installed in their tenant. But that might just be me not understanding how this all works.

This is interesting. Without the app installed in a tenant, is this data even accessible to the customer?

#10

Are you sure you want the answers to these? :smile:

You can uninstall an app and continue to have a subscription. Some customers make use of this to either remove apps temporarily while they troubleshoot functionality and/or if they’re not ready for something they’ve built/integrated isn’t quite ready yet on an instance. It can also happen when an administrator doesn’t have billing rights on a system (which happened to me the other day) - uninstall the app as an admin - the billing person has to remove it.

As far as the content properties - I’ve been asking this for a while now and from what I know/been told:

  • Issue properties are available for anyone that can access the issue (this involved editing/deleting).
  • User properties are available for admin and the app that creates it (or so I’ve been told by jira devs) - there’s no documentation supporting this).
  • Add-on properties are available for admin and the app that creates it (this is documented).
  • I have no clue about the properties - but I’d imagine that they’re like the Issue Properties)

But all of that said - there’s nothing stopping me from doing an export from a jira system and getting all of the properties from the zip file.

1 Like
#11

We would also be very interested in this feature. Mostly, we would like to get a webhook whenever a Jira instance is terminated or (as @danielwester mentioned) restored.

I would suggest all the interested vendors to also vote on this issue: https://ecosystem.atlassian.net/browse/AC-863 as this has been in the backlog for quite a while now.

#12

I have build a cleanup system around the data available in the marketplace. I run a daily job to update all the SENs of the known tenants of the app. And have another daily job that will query the marketplace for SENs of licenses that where inactivated 180 days ago and will delete the tenant data but not the Atlassian host information as this is required in order for a future install event hook to be properly processed.
There are some tenants that don’t have a SEN and I have not yet developed a good solution for this, but at least with these two daily jobs, I can cleanup tenant data 180 after yet uninstalled the app. This also gives then time to 're’start where they left off when they reinstall the app.

#13

Encrypt all data at rest where decryption is linked entirely by both a client secret and one you have linked to their licence.

No licence and they have no access to the data and you don’t have liability of access to their unencrypted data.

Treat a decrypted cache layer as “in transit” if you need to improve latency. Cache lasts X hours.

Doesn’t absolutely remove data storage on your systems, but you could have a retention policy to suit your/customers best case.

1 Like
#14

Unfortunately we don’t have a single unique shared secret that we can encrypt by. Otherwise that would be awesome in the short term (I would still need to delete things at some point).

#15

I guess this was a moving forward from here point.

It’s been a technique we’ve been applying for PI data for a while. Lends itself well to auditable access ledgers also.

1 Like
#16

Anyone at Atlassian that have some suggestions?

1 Like
#17

With the latest set of announcements and the focus thing. @WarrenChen @nmansilla is it possible to get somebody from Atlassian to dig into this thread? I am kinda surprised that after 2 months - there hasn’t been an answer from Atlassian to a key question about how to build on Cloud…

1 Like
#18

Daniel, we recognise that there are issues in this area of app lifecycle management.

Data Retention

https://confluence.atlassian.com/cloud/data-storage-faq-873871367.html discusses data retention in general, from Atlassian’s perspective:

After you have been unsubscribed, Atlassian retains your data for 15 days (for evaluation sites) or 60 days (for paid subscription sites).

So, based on that, I would recommend the approach:

/uninstall with a delay

I would recommend the delay be 60 days for paid apps. Free apps could probably go with 15 days but 60 days there too, may be prudent. Whatever you choose, this should be made clear to customers when they install your app.

Instance Deletion

Currently instance deletion does not send uninstall events to installed apps. We know this is an issue in the Connect framework. This is primarily due to historical architectural reasons. The Connect framework runs as a plugin with the Jira/Confluence product instance. Normally prior to deletion an instance is suspended. It is no longer running due to conditions such as non-payment. The site may be reinstated during this period, once payment is successful, for example, so the site’s state (including addons installed) is preserved.

When a site transitions from suspended to deleted, we do not reactivate the site prior to deletion. As a result the Connect framework is not active to uninstall the site’s current apps.

We are moving to a more centralised model of app installation tracking, where the uninstall actions will not require the site to be active to deliver these events. When this change is made, site deletion will be able to internally trigger app uninstalls. We are not there today and I do not have a timeframe to give you for when this change will occur.

So in lieu of an event sent at the time of instance deletion, I would recommend the polling for access approach. I recommend the data retention times would start at the point you decide a site has been deleted.

How could an app achieve this? All apps are different so we can’t give a definitive statement about how to implement this data expiry. The following approach is one possibility:

Each tenant record could be tagged with a last-accessed data, which is updated whenever there is an interaction with the instance. All instances with no active interaction in the last week would be polled. Polling would recur at regular intervals (daily or weekly, say). At 60 days since last active date, the site would be considered as deleted. At 120 days the instance’s record could be removed.

3 Likes
#19

What endpoint should be used for polling and what HTTP responses / status codes indicate a deleted instance?

#20

Polling for the license end point would work. If your credentials are revoked then you’ll get a non 2xx error. Note though you will still have access even if the customer ends your license but haven’t uninstalled the app yet… Happens a lot.