Changes to the installation of DC apps

Announcements
,
1

To enhance security, we’re changing the way Data Center products handle app installation using Universal Plugin Manager (UPM) and REST APIs. App installation through UPM is now disabled by default. System admins can still enable app installation.

This change doesn’t affect the installation of apps from the Atlassian Marketplace.

When will this change be available?

Upgrade to the following product version, to activate this change:

What changes for admins?

Changes to the user interface

We’re removing the Upload app button from the Universal Plugin Manager, so system admins won’t be able to install apps through the Administration.

Here’s an example of the Bitbucket user interface before and after the change.

Before

manage-plugins-before
manage-plugins-before2136×744 60.4 KB

After

manage-plugins-after
manage-plugins-after2127×725 59.1 KB

Changes to the REST API

We’ve changed the /rest/plugins/1.0/ REST APIs to block the possibility of installing apps.

How can admins enable API app installation?

The system admin can enable the functionality by setting the upm.plugin.upload.enabled system property to true .

:information_source: To apply changes admin needs to restart the product instance.

For further details, check respective release notes.

:information_source: App installation will be disabled for Data Center product instances running on Atlassian Marketplace Plugin Suite (AMPS). To enable it you need to switch to developer mode or set the upm.plugin.upload.enabled property to true.

Is there LTS support for this functionality?

This change will be also implemented in the following supported LTS versions:

  • Jira Software 9.12 LTS and 9.4 LTS
  • Jira Service Management 5.12 LTS and 5.4 LTS
  • Confluence 8.5 LTS and 7.19 LTS
  • Bitbucket 8.9 LTS
  • Bamboo 9.2 LTS
1 Like
2

Hi @CharafEddineSAIDI ,

thank you for letting us know.

I think we/I would highly appreciate knowing about such changes before we spin up the latest instance in our dev environment and coworkers are asking where the button went…

For Confluence, the change was communicated before, but for the other products was quite a surprise, especially because it is only a new minor version.

And when talking about major and minor versions, I would expect such a change to be in a major version…

11 Likes
3

It’s a shame this is a retroactive announcement, and no RFC was put forward for this change. The actual security benefit from this change is minimal, at least for any internet connected instance.

Various apps from marketplace can be installed that allow access to the underlying host system, DC is by no means a security sandbox.

It would have been nice if in addition to this the UPM supported downgrading app versions via the UI. Or if the UPM supported upgrade/downgrade paths elegantly at all.

This is one of the primary reasons our customers would upload a jar directly, when prompted to do so by our customer support team.

10 Likes
4

That’s a great point @rlander !
@CharafEddineSAIDI can you please provide detailed steps on how an installed plugin can be downgraded without causing an outage of the whole DC cluster?

2 Likes
5

Just bring in the discussion from the Jira “suggestion” https://jira.atlassian.com/browse/JRASERVER-77129 as it has already a few complains on that “security feature”.

1 Like
6

Well if it’s multi node DC you could just roll one node at a time, it’s a system property so should impact only that node.

Still a major inconvenience for impacted customers.

1 Like
7

Can you load a downgraded JAR with a rolling restart?

1 Like
8

Something to test, I wonder if the app replication still works if only a subset of nodes allow uploads.

We have a ScriptRunner script on hand that can flip the switch without rolling a node, putting that out into the public seems in opposition with Atlassian’s intent though.

1 Like
9

upm.plugin.upload.enabled doesn’t work on jira 10 EAP01

1 Like
10

Atlassian puts note on cookie jar that says “Don’t eat!”
Atlassian wonders why all the cookies are gone.

That’s security!

1 Like
11

Yep, this one was a shock to me too. Lets say ‘great job on the comms’. Sigh.

1 Like
12

If I’m not mistaken customers are forced to install the latest app version compatible to the host product. That should be the default, of course.
@CharafEddineSAIDI - are there any concerns with adding a version selector to the install modal? This way a customer could uninstall the latest app version and re-install the version they need to downgrade to, if they are told so by support. The proposed version selector should be minimal in appearance :smile:
This solution should cover the requirements of both sides:

  • (1) no uploads [Atlassian DC security]
  • (2) app version downgrades [Vendor support]
    Thanks for your consideration.
5 Likes
13

Hi @CharafEddineSAIDI

A few weeks ago we talk about Jira 10 with JAVA 17 and Jira 9.12 with JAVA 8/11.

The answer by @MarekTokarski was :

Now I would like to understand how we can follow your recommendations and at the same time remove the “upload” button from UPM by default ? How will the client be able to choose the most adequate version to their JIRA? Each client will have to enable API app installation ?

An idea: maybe we should tag the version of our apps on the marketplace to specify if it is more suitable for Jira 9.4/9.12 or Jira 10?

5 Likes
14

Hi @CharafEddineSAIDI ,

This change will significantly affect us, particularly our support team. Here are only few scenarios where manually uploading a specific version of the app is necessary:

  • Providing a custom (snapshot) app version with some additional logs to a client for debugging complex support ticket issues.
  • The inability to roll back to a specific version
  • Upgrading to major app versions with breaking changes that require multiple steps.

Given that it will no longer be possible by default to offer this option to impacted customers, for each of the above examples customer would need to enable this property meaning restart each node on production at least twice (to enable and at the end to disable it). Do you agree with this?
If yes - it is highly impractical and will likely extend the duration of open support tickets with impacted customers.

It would have really helped if we had been aware of this change before releasing it in order to prepare our team.

Dusan

10 Likes
15

Hi @chrschommer

We have prioritized this as a security fix. Since it is not a breaking change, we have planned to release it as soon as possible in the form of a bug fix version.

I agree, we could have communicated changes across the products earlier and we’re working hard to make the communication efficient and transparent.

Please also note that we’re increasing efforts to make our DC products even more secure and performant. You can expect us to ship our releases more frequently to enhance our security posture.

1 Like
16

Hi @rlander

Among the recent efforts we have made in terms of security, updating the security bug fix policy is one of our key priorities. While it has always been strongly recommended for our customers to be on the latest version, we want to emphasize this message even more.

Therefore we don’t recommend any workarounds to downgrade apps, plugins, and libraries.

2 Likes
17

Hi @CharafEddineSAIDI,

Since it is not a breaking change, we have planned to release it as soon as possible in the form of a bug fix version.

May I ask what criteria you use to decide what is a breaking change?

As the Head of Customer Support, I can say without hesitation: this IS a breaking change for any Marketplace Vendor. Many customers can’t restart a node on demand, just to facilitate troubleshooting.

Instead of using a system property to change this behavior, would it be possible to move this option within the UPM settings?

Thanks,
Nicolas.

5 Likes
18

That sounds good in a vacuum, but reality is a lot more complex.

Customers will need to downgrade apps, if I ship a critical bug to marketplace and then retract that version, customers will need to downgrade. Or should we leave them with a broken instance in the name of security?

I will reiterate, the DC platform is fundamentally insecure when apps are present on the instance, there is effectively no sandboxing of apps, they can do almost anything the JVM process can.

This change does not make the platform more secure, if I am a bad actor I can install one of many apps from the marketplace that would effectively give me backdoor access to the system.

Would this change not have made more sense to be implemented once the marketplace and UPM support signed apps? I can understand blocking uploads of unsigned apps by default, however this change in its current form is rushed.

6 Likes
19

Hi @CharafEddineSAIDI

thanks for your reply.

I think we don’t have the same opinion regarding this being a breaking change or not. Like the other marketplace partners in this thread, I see this differently.

From my experience in support, if you’re app breaks and customers cannot simply downgrade, this results in huge problems. I have talked to customers who need a two-week window before an instance restart. It is not that easy in practice. Think about it, Atlassian DC aims for big multinational companies. They cannot simply restart Jira while people on 3 continents in different time zones work.

And downgrades will and do happen because regardless of all the unit, integration, or end2end tests, things go wrong. Especially in a highly complex environment like it is for Atlassian products.

Apart from downgrades, we offer a support tool for our app as a separate non-marketplace app. This cannot be installed easily anymore now. While we now want to get this into the marketplace, we need to spend a considerable amount of resources to make this happen. We have to rewrite things, and clean up features because when this app is in the marketplace, anyone can install it. Before, we pointed customers specifically to the app and worked together with them on the case.
Now, anyone will be able to find the app on the marketplace and install it, and potentially use it wrong.

I also do not understand why you are forcing this change so fast.
Instead of announcing this months before such that we could react before the change (e.g. by working on getting our support tools in the marketplace), customers upgrading to the latest Jira/Confluence/etc will have bad luck. Because if something goes wrong, they cannot simply install the support tools or downgrade anymore and might need to wait weeks before a window for restarting the instance.

And don’t get me wrong. I totally understand that you want to strengthen security. However, why do you not introduce signed apps instead? With such an approach, only vendors controlled and authorized by Atlassian would be able to create and distribute apps.
With signed apps, the upload app button does not need to go because you can only install signed apps. We could simply sign our support tools, and customers would be able to install them. Downgrades would still be possible because the old versions are still signed.
While I don’t think adding signatures to apps solves all problems (and if we have a look at the Apple ecosystem we can see the problems it introduces), it would be a compromise between security and flexibility for our customers.

Best regards,
Christopher

7 Likes
20

Working in a Support department for a Marketplace vendor, this evolution is problematic as we will no longer be able to ask customers to downgrade/upgrade our apps as easily. This will significantly slow down our investigations. :-1: :-1: :-1:

4 Likes