Forge Storage data persistency behaviour

[remie]

Hi Atlassian,

We need to talk about data persistency in Forge Storage. Like, have a good talk. A real talk. A talk that will help both you and the Atlassian Marketplace Partners to continue our journey on making Forge the best platform to develop apps.

Because you are doing something really weird with Forge Storage, something that is really not in line with Forge being a FaaS platform.

Here is the thing: for better or for worse, Atlassian Marketplace partners are independent companies. You designed it that way. You also create a legal framework in which you persistently claim that we have a one-on-one relationship with our customers who buy our apps.

Atlassian is just acting as a reseller, offering a shop window (Marketplace) and payment handling. But once the customer installs the app, they enter into an agreement between themselves and the Atlassian Marketplace Partner. Apart from any licensing & billing issues, Atlassian does not accept any responsibility of what happens with regard to that relationship.

As a Marketplace Partner, this gives us a lot of freedom. Freedom to operate the way we want, to choose whichever technology stack we prefer. You even gave us the tools to do so: Atlassian Connect and Atlassian P2 are both frameworks that allow us to create the apps the way we see fit (with P2 apps having the sole requirement that they are compatible with the Java version and database engine on which the host application runs).

For Atlassian Cloud, this means that we can choose whatever hosting provider we want. We can use a $5 VPS or Digital Ocean droplet. We can go all in on Kubernetes. We can use FaaS/Serverles solutions like AWS Lamba, Firebase, Cloudflare workers or Vercel.

You don’t dictate any of that because this is our choice, and it is something we have to communicate with our customers.

Now with Forge, you’ve created a nice in-between solution: it is a FaaS like any other, but operated by Atlassian. This is truly a unique selling point (apart from the fact that it is still free ). Running our application code as close to the Atlassian host product as possible, enabling the ability to keep data contained to Atlassian systems is truly a huge leap forward with regard to improving the security posture of our apps.

But… (there is always a but)

You seem to forget that from the “legal” perspective, Forge is just any other provider. It is similar to AWS, Firebase, Cloudflare, etc. It is a FaaS offering a service, and we, as Marketplace Partners are your customers.

We are running our business on your platform, to fulfil our contractual obligations to our customers. Our customers ask us to operate our app, handle their data and store it.

This is not your data. It is the data provided to us by customers, as a result of a legal agreement between the customer and the Atlassian Marketplace partner. And we, as a Marketplace Partner, enter into a separate 3rd party agreement with you, Atlassian, to host that data for us, similar to how we enter into agreements with other providers like AWS, GCP or Cloudflare.

The fact that you make decisions on how to deal with this data in Forge Storage is giving really icky vibes that shows that you do not fully understand this very important legal fact.

It already started with the fact that Atlassian coupled data residency of Forge Storage with data residency of the host product. Without any notification to the Atlassian Marketplace Partner, Atlassian moves data from one region to the other. This is really disturbing, because if you fuck up, WE are responsible. How can we be legally responsible for an action that you take without our explicit approval or explicitly requested by us?

But it becomes an even bigger problem when you read this:

Following an investigation, we have identified that:

• There is no data loss.
• A change was made as a part of the Forge data residency rollout, where data is now bound to the installation identifier instead of the app and site.
• There was a gap in our communication with partners around this specific change.
• The existing behaviour for app uninstallation, where data persisted between installations, did not match what the user interface told administrators. I.e. “Uninstalling will permanently remove this version of the app from ”.
• The new behaviour of not persisting data between installations is in line with the intended experience.

As a result of the investigation, our next steps are that:

• We will define and include new documentation on the Forge app installation lifecycle to clarify the expected behaviour and communicate this with partners.
• We will explore options for restoration of data from previous installations directly in the admin user experience.
• We will enable support to re-link data from previous installations as a part of the ECOHELP process within 14 days from app uninstallation.

(Atlassian Developer Status - Forge Storage data inaccessible after re-install of app)

This is really problematic.

You have decided that you can dictate when it is appropriate to delete Forge Storage data on behalf of the Atlassian Marketplace Partner.

By doing so, you are directly interfering with the legal agreement between the customer and the Atlassian Marketplace Partner, but without taking any of the legal responsibility for data loss.

Data retention is not your responsibility. It is the responsibility of the Atlassian Marketplace Partner. Image having AWS deleting RDS database data or GCP emptying a Firestore database without prior notification!

You do not get the angry customer, we do. You will not be held liable, we are.

The sole purpose of Forge Storage is to solve the problem of data egress. By adding opinions about data residency and data retention, Atlassian is injecting itself into a legal relationship which it so carefully tried to avoid.

I’m really asking you to reconsider your attitude towards Forge Storage and ask yourself the hard question to determine what role you are playing here, for the legal ramifications of your current course of action makes it very dangerous for businesses to take on the risk Forge Storage imposes on their operations. I for one will not be able to justify using Forge Storage from a corporate risk management perspective, as I do not wish to be legally responsible for whatever whim Atlassian operates on.

CC: @tpettersen

[markrekveld]

Thanks @remie for taking the time to write this.

Ever since Forge was announced I have been looking if it would be a good fit for me to move to, but time and time again I see issue popping up. This being the biggest one by far for me.

I truly think Forge can be better then Connect, but with Marketplace Partners having no control over the running environment but have all the responsibilities for the apps they operate on it makes a a very risky step to make. For me it a risk I cannot take.

I’m also wondering how the stands of Atlassian with Forge impacts the ISO27001 and SOC2 certifications for Marketplace Partners.

[danielwester]

Thank you @remie for bringing this up. It is really important that Atlassian manages the data based on the appropriate data relationship agreement. This is really a blocker for us from adopting Forge storage (or compute) due to the legal issues that this raised.

/Daniel

[JulianWolf]

I second the idea to have a discussion about the overall contractual relationship between Forge, partners, and our common customers, especially with regard to the Forge Storage API. Thank you, @remie, for that.

Besides that, I’d like to ask Atlassian to take immediate action regarding the current incident with the Storage API. Removing all data when a customer uninstalls (and reinstalls) an app is not an option. I understand that it is an unlinking and not actual data loss. From a customer perspective, it doesn’t matter. Data loss takes place at the latest as soon as a customer continues working on the emptied Storage API, which then needs to be re-linked via support to rescue the old data. Just think about a scheduled job or events that occur automatically and write crucial data in the Storage, which cannot be merged with the old data.

Rolling out a change like that without prior notification is a problem!

I truly like the idea behind Forge. However, the current situation has made us stall our ongoing Forge migration for one of our largest apps.

Please immediately take action to keep the data (for 30 days or another appropriate duration) after an app uninstallation. Let’s have a proper discussion on how to handle this in the long run.

[JamesDumay]

Hi Remie,

Thank you for your honest perspective on data handling, we genuinely value the input from all our partners.

One of the main reasons we created the Forge platform was to better support our developers in meeting customer legal and compliance requirements. That is why we have designed Forge Hosted Storage to streamline the technical process necessary for our partners to build an app that can follow customer instructions around data storage issues like data residency location and deletion. Our customers have expressed a desire for data controls for our products and for Marketplace Apps and in turn, many of our developers have requested that we create features to better enable them to meet these customer expectations.

We understand the importance of flexibility in choosing a storage platform, and we want to assure you that it is entirely up to you whether you choose to use Forge Hosted Storage or a different storage option. We believe in empowering our partners to make the best decisions for their needs.

Atlassian is a global entity and as such has obligations across countries, jurisdictions, laws and compliance standards on how we need to handle our customer’s data. You can find a description of these and more on the Forge Privacy and Security FAQs page.

We are confident in our approach, however we recognise there will be differences in viewpoints. We understand that due to the evolving compliance and regulatory landscape partners and customers may have different needs and perspectives regarding data management. For this reason it is critical to our mutual continued success that we keep an open mind and work together to better understand each other’s viewpoints in order to improve the Forge platform.

We deeply value our partnership with you and are committed to understanding your needs. Collaborating to improve our platform is a top priority for us, and we appreciate the feedback you’ve provided. Again, we hear you on this issue and will be contacting partners as it evolves.

Thanks,
James Dumay
Group Product Manager, Forge

[PhilipFeldmann]

Sorry but someone needs to say it: This is the most underwhelming and disappointing chat-gpt answer I’ve seen in a long time.

We are confident in our approach, however we recognise there will be differences in viewpoints.

How can you be confident in a approach that hurts customers, hurts vendor businesses and ultimately hurts Atlassian What happened to “don’t f*** the customer”? How do you expect customers to have confidence in your platform when their data is gone after they’re forced to re-install a business-critical app because of a UPM or licensing bug. How do you expect vendors to have confidence in the ecosystem when you change a critical part of the platform you’re pushing so much without any notice period?

[remie]

Hi @JamesDumay,

Thank you for your thoughtful response.

But it still feels like we’re not on the same page. I guess we should dive into the legalities! Let’s start with the obvious:

Atlassian Marketplace Terms Of Use
Section 3 - Use of Marketplace Apps
3.1 Vendor Terms - (a) Third Party Apps

Third Party Apps are subject to the third party’s Vendor Terms, not the Atlassian Terms. By ordering, installing or enabling any Third Party App, you are entering into the Vendor Terms directly with the applicable third party Vendor. Atlassian is not a party to, or responsible for compliance with, any third party Vendor Terms, and does not guarantee any third party Vendor Terms are adequate for your own needs.

Can we agree that this means that as soon as the customer installs a Third Party App, they enter into agreement with the vendor and not with Atlassian?

I’ll give you a hint!

Atlassian Marketplace Partner Agreement
5. Your Content; License to Atlassian; End User Licensing
5.6. End User Terms

You, not Atlassian, license your Apps to end users, and you must provide your own End User Terms and End User Privacy Policy with any Marketplace App. Your End User Terms and End User Privacy Policy must comply with, and be consistent with, the terms and conditions of this Agreement, including Section 8.4 (End User Data and Privacy-Related Obligations). You agree that Atlassian does not and will not have any responsibility or liability related to compliance or non-compliance by you or any end user under the applicable End User Terms or End User Privacy Policy.

So now that we have established that Atlassian considers that the customer enters into a direct agreement with the Marketplace Partner, and waivers any liability as a result of that agreement, let’s look at what Atlassian thinks about data collection & use of End User Data by third party vendors!

Atlassian Marketplace Terms Of Use
Data Collection and Sharing
4.2. Third Party Vendor Use of Data

Atlassian is not responsible for any access, use, transfer or security of data or information by third party Vendors or by Third Party Apps, or for the security or privacy practices of any third party Vendor or such Vendor’s Third Party Apps and third party processors. You are solely responsible for your decision to permit any third party Vendor or Third Party App to access or use data to which you’ve granted access.

Can we agree that this means that Atlassian does not consider itself liable for any data processed/stored by Marketplace Partners? Guess what, the answer also lies in the Marketplace Partner Agreement!

Atlassian Marketplace Partner Agreement
8.4. End User Data and Privacy‐Related Obligations.
(b) Collection and Use.

Atlassian shall not be liable for, or have any responsibility in connection with, End User Data processed by you or your App, and such activities with regard to End User Data are not in any way by or on behalf of Atlassian.

Ok, so we have established that according to the global entity Atlassian, customers enter into a separate agreement with third party vendors and any data collected, processed and stored is done by the third part vendor as a direct result of such agreement and is explicitly not done by the third part vendor on behalf of Atlassian.

Atlassian has made it explicitly clear that she is not a party here, so that means that this part:

Atlassian is a global entity and as such has obligations across countries, jurisdictions, laws and compliance standards on how we need to handle our customer’s data

Is just really not applicable, now is it? It’s not your customer’s data. It’s the third party vendor customer’s data.

It becomes even more interesting if we dive into the specifics of Forge and Forge storage. You mention that Atlassian created Forge based on customer request:

Our customers have expressed a desire for data controls for our products and for Marketplace Apps and in turn, many of our developers have requested that we create features to better enable them to meet these customer expectations.

Sounds great. But your customers aren’t the customers of Forge, right? Because whenever we publish a Forge app to the Atlassian Marketplace, we enter into an agreement with Atlassian. And the Forge Terms not only comes with a reference to the Developer Terms, which are eerily similar to the Marketplace Partner terms when it comes to data processing, it also has a reference to the Forge Data Processing Addendum which is accepted upon accepting the Forge Terms.

And that Forge Data Processing Addendum is perfectly clear about the responsibilities when it comes to handling data:

Relationship of the parties: Where Applicable Data Protection Law provides for the roles of “controller” and “processor”:

1. (a) Where Atlassian processes End User Personal Data on behalf of Developer in connection with the Services, Atlassian will process such End User Personal Data as a processor or Sub-processor on behalf of Developer (who, in turn, processes such End User Personal Data as a controller or processor respectively) and this DPA will apply accordingly. A description of such processing is set out in Annex 1(B), Part A of Exhibit A to this DPA.

So let’s go back to this statement:

Atlassian is a global entity and as such has obligations across countries, jurisdictions, laws and compliance standards on how we need to handle our customer’s data.

I guess we can conclude that this is not applicable to Forge, Forge Storage and third party apps in general.

But it does highlight the exact issue that we are trying to discuss here. Atlassian is pretending that is has obligations towards customer data. It is ingrained into the very DNA of your company. Even though your own legal frameworks are very much explicitly telling you this is not the case.

Forge and Forge Storage is just like any other hosting platform a third party vendor can choose to use for servicing their customers, but Forge & Forge Storage is the only platform that wrongfully feels that it get’s to be opinionated about how this data should be handled.

And that needs to stop. Or you need to change your legal agreements.
As always, Atlassian can’t have her cake and eat it

[danielwester]

Again… What @remie said…

Atlassian keeps getting their DPAs mixed up - and this is a very dangerous situation.

There’s mainly 2 seperate chains.

Customer (Controller) → Atlassian (Processor) → Any other entity that Atlassian makes use of to provide Jira functionality.
Customer (Controller) → App Vendor (Processor) → Atlassian Forge (Processor)-> Any other entity that Atlassian makes use of to provide Forge functionality.

Making these chains murky starts heading into a dangerous territory in case any partner gets audited. Especially if data comes back etc.

It is crucial that the vendors are in charge/aware of data retention etc. Associating it with the host instance (which is under a different DPA agreement) makes difficult to argue that the App Vendor told Atlassian to do something on our instruction…

[scott.dudley]

Hi @JamesDumay ,

I echo what Philip wrote above. To expand on this further, while there are certainly what appear to be very real legal issues related to the discussion, I think that the customer experience issue is just as important. Is there anyone here from Atlassian (@ibuchanan ?) who might be able to share (or suggest the invitation of other Atlassians to the discussion who can share) a broader, ecosystem-wide perspective on how customers expect their data to be handled?

The whole concept of “we are going to vaporize all customer data as soon as someone hits ‘uninstall’ on your app” is baffling. I admittedly do not have an iron in this fire since I am not using Forge, but this type of scenario is scary and it makes me reluctant to consider using the platform in the future.

Here is the existing behavior across different data types when a customer uninstalls an app:

Data type – Is customer data automatically deleted on uninstall?

Server/DC app data – NO for any serious vendor
Connect apps – NO for any serious vendor
Connect app properties - NO
Content properties - NO
Forge hosted storage - YES

The recent incident was resolved by citing a piece of text in a UPM dialog, presumably written by some random engineer a decade ago, that says that “Uninstalling will permanently remove this version of the app”.

We all see the outlier in the table above. In general, nobody reads dialog text, and especially not when things have been working a certain way for decades, so what the text says is mostly irrelevant. Atlassian must also surely understand that the deletion of customer data, when such deletion is not intended, can be catastrophic.

Your note implies that our mutual customers are actually asking you to delete data this way. If this is what your data shows, then I would guess that you are probably asking the wrong question. “Do you want to be able to permanently remove a third-party app and all of its data from your site?” is a legitimate question, but “Do you also want Atlassian to automatically remove an app and permanently delete all of its data as soon as you perform a quick reinstallation for administrative purposes?” is a question which should have a completely different answer.

I think you could make most people happy if you just implement a 30-day deferred deletion procedure, which is reset upon reinstall of the app. If you have customers who are adamant about deleting data Right Now, then give them an “Obliterate All” button that they can press. And if you truly can’t do either, then you should really add a couple of blinking red 72-point font warning dialogs that pop up when uninstalling an app which say that “when you perform this action, all app data will be forever and irrevocably deleted”, and make people click it six times (or whatever).

[tbinna]

I agree with everything that’s been said here and also want to reiterate the thought of why not just give the customer the choice to delete all data or have it deleted after a retention period (in case they reinstall). Another benefit of this solution is that if the customer explicitly wants a clean reinstall, they can choose the option to delete data upon uninstallation. The most important point is that the decision is given to the customer.

The current method seems just a brute force measure to fit some compliance criteria without much thought on the further implications or better solutions. This also shows in the way the behavior has been hastily adjusted without notice.

[dusan.spaic]

@JamesDumay could you please let us know what changes you will add to the Forge documentation and inform us when the documentation is updated? It is important to be very clear about the retention period of the customer data depending on the different statuses that the app/license may encounter, possible backups, or retrieving the data. We need precise information so that we can be very clear with our clients and our policies defined by different compliance standards.

thanks
Dusan

[chhantyal]

Atlassian, you decided to make sweeping platform changes without any notice to vendors? Where is this documented?

Comically, Atlassian support that handles licensing related issues is as confused as us vendors and customers

[chhantyal]

This just happened:

Atlassian support uninstalled and reinstalled our app in a customer instance to resolve a licensing issue.

The licensing issue is now resolved, but all the data is wiped out. The customer can’t believe what just happened!