RFC-39 Allowing installation of only signed apps on customer instances

[MalathiVangalapati]

RFCs are a way for Atlassian to share what we’re working on with our valued developer community. Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Summary of Project:

In this project we’re exploring options for app/plugin signing to ensure only the signed apps are being installed and seeking input from this audience to brainstorm potential approaches

• Publish: 12 Mar 2024
• Discuss: 26 Mar 2024
• Resolve: 10 Apr 2024 19 Apr 2024

Problem

Security and trust is foundational to everything we do. We’ve identified opportunities to continue to strengthen our security posture with regards to ecosystem. One clear opportunity identified has been preventing persistence through access to a SYSADMIN account. The risk associated with persistence lies in the ability for the attacker to maintain access to the system even after initial vulnerabilities have been addressed, making threat removal more challenging.

To, we’re exploring options for app/plugin signing and welcome input from this audience to brainstorm potential approaches.

Proposed Solutions

During this process we’ve explored several short-term and long-term solutions, including but not limited to:

1. Disabling custom non-marketplace plugin uploads by default at a file system level.
2. Limiting plugin upload capability to SYSADMIN operations across all DC products.
3. Implementing IP address allowlisting for SYSADMINs to prevent plugin installations over the public internet.
4. Requiring 2FA for SYSADMINs.
5. Restricting DC user account privileges to prevent execution of web shell commands.
6. Implementing integrity and authenticity verification measures (e.g., code signing, certificate chain verification) for installed plugins in DC applications.

After reviewing the mentioned solutions, we’ve chosen to delve deeper into exploring the preferred option of app signing.

App signing

App signing offers several advantages, including enhanced security, trust, increased user confidence, protection against tampering attempts, and compliance with security standards. These benefits make app signing an essential practice for app developers and for Atlassian.

This document initially presents the requirements and then provides a detailed explanation of the various approaches.

---

Requirements

Air-gapped instances support	The solution must allow air gapped instances to install and verify plugins offline.
Backward compatible	The solution should be compatible with existing instances that have unsigned plugins already installed.
Custom plugins support	Clients must be able to install their custom plugins.
Certificates management	Certificate management should be made simple for both our App vendors and Atlassian.
Extensible	The solution must be extensible to support additional features, such as preventing the execution of unsigned code.
Encryption Algorithm	To comply with Atlassian Security policies and local country legal constraints (e.g., FedRAMP), we recommend using EdDSA as the signing algorithm for the first implementation phases.

---

Native Approach OPTION 1

This solution relies on native JDK tools to sign and verify plugins. It has several benefits, such as embedding digital signatures in the same archive as the source code, which reduces the need to manage additional files. Moreover, it eliminates the need for external services to sign or verify plugins. However, it has a drawback when it comes to managing keys and certificates, which should be fully provided by Atlassian.

Advantages	Disadvantages
Phased approach	Keys and certificates management
No external dependencies	Rely on JDK binaries (ie. jarsigner)
Signatures embedded in the same JAR	Parsing jarsigner output may be challenging

---

Sigstore Approach OPTION 2

This approach leverages Sigstore to sign and verify plugins. Using an OpenID Connect (OIDC) provider to generate ephemeral keys for signing plugins eliminates the need for key or certificate management. However, external services such as a Transparency Log (Rekor) are critical to verify signatures.

Advantages	Disadvantages
No keys or certificates management	New critical service creation/maintenance
Innovative approach (presented as the new standard for OSS)	External dependencies
Simple signing for apps vendors	“bundle” files management
	Complex architecture

---

Bouncy Castle approach OPTION 3

This solution is similar to the Native approach. It simply relies on a 3rd party library to sign and verify JAR files. This is the simplest solution we can think of when it comes to signing artifacts.

The main advantage of using a third-party library lies in the ability to fully control the signing process. This might help restrict signing algorithms, for example. On the other hand, Atlassian will need to provide all the necessary services to manage key pairs as in the Native approach.

The Bouncy Castle library has been used here but it may be replaced by another library.

Advantages	Disadvantages
Phased approach	Keys and certificates management
Full control of signing/verification steps	signature files management
Strong community and OSS support

Actions

• Do you believe that implementing an ‘app signing’ feature for apps would reduce our attack surface and enhance our security posture?
• Please provide feedback on the preferred option and explain why you believe it’s the preferred choice.
• What specific features or enhancements to this solution would facilitate its implementation and effective management?
• If this feature were to be introduced, is your organization equipped to meet the additional requirements for implementing these changes? What assistance and support would you require to better adopt this feature?
• Could you outline the current process for ensuring the security of your apps?
• Feel free to share any additional comments or thoughts on this feature.
• If you are interested in a follow-up 1:1 with our team to share or know more, please leave a comment on this post and we will get back

[jens]

@MalathiVangalapati

If this feature were to be introduced, is your organization equipped to meet the additional requirements for implementing these changes? What assistance and support would you require to better adopt this feature

This probably depends to a large degree on how Atlassian supports the signing for example through AMPS / plugin SDK.
What are your plans here?

Thanks,
Jens

[scott.dudley]

What specific security problem are you trying to solve with double-signing, or more specifically, by asking vendors to sign their own plugins before submission?

The DC plugin tooling is already in a neglected state (a broken default banned dependencies configuration that doesn’t work with current versions of any product, REST javadoc generation not working, lost signing keys, and a Maven version that was broken for a few years up until recently) and I would be wary of cramming more requirements on the vendor side.

How about a magic solution #4 where vendors upload the .JARs to the Marketplace after authenticating themselves with your preexisting 2FA solution, Atlassian performs whatever magic “blessing” it needs to do, and spits out a link (via UI and REST) to the signed .JAR?

You sign it however you want, you verify it in whatever way suits Atlassian best, and it does not create additional workload for vendors.

[jcarter]

Please provide feedback on the preferred option and explain why you believe it’s the preferred choice.

I think options 1 or 3 would be critical for anyone operating an Atlassian app behind a firewall, which is a non-trivial set of customers. My hunch is that Bouncy Castle is probably the better of the two, since there’s a strong community behind it.

Do you believe that implementing an ‘app signing’ feature for apps would reduce our attack surface and enhance our security posture?

I mean, it would add a checkpoint to one potential attack vector, but that checkpoint is already effectively guarded by “have admin access to the system”. I think the most meaningful added security benefit here is preventing tampering with plugin files in transit.

If this feature were to be introduced, is your organization equipped to meet the additional requirements for implementing these changes? What assistance and support would you require to better adopt this feature?

This would present a real problem for customers using Script Plugins. We would either need to guide them through the necessary process on how to sign their apps OR completely re-implement Script Plugins to not use UPM at all. The good news is we aren’t busy at all with the looming Grey API removal or Platform 7 upgrades, or anything similar coming down the pipe from Atlassian. Haha! Just kidding! We are!

Another real problem is that it potentially breaks the downgrade path, which is necessary for those rare cases where an app vendor ships a bug and a customer needs to downgrade. There would at least need to be some kind of warming period where app signing was available, but not required.

[steve.behnke]

We have some 50 plugins between Jira, Confluence, Bitbucket, and Bamboo that we’ve written, entirely internally at SpaceX. In cases like Bitbucket with Client Side Extensions, we even help some developers at SpaceX create their own extensions. So I’d expect that we’d want to be performing this operation entirely internally if at all possible. Today we mostly just get it in Bamboo and hit it with atlas-package so, something simple would be good…

[tobitheo]

Let me start by saying I appreciate the effort on this. Parts of the vendor community has been asking for this for some time now and it’s good to see this RFC. We have discussed internally how this feature could be implemented in the past and we have been anticipating your solution proposal(s). This post is bit of a discussion and stream of my thoughts on the matter. I am rambling a bit, I’m afraid, please forgive me. This is also my opinion on the topic and does not necessarily represent the opinions of resolution or my colleagues there

The question

The big question I see is this: What threats exactly does this try to defend against? I suppose it would be vendor marketplace credentials (including 2FA codes) getting into the wrong hands and a malicious actor uploading bad versions of apps infected with malware. And maybe also Atlassian servers where the MPAC artifacts are stored being compromised?

Options 1 and 3 fail at the single point of compromise issue because if Atlassian’s signature service is compromised, it’s game over.

Option 2 fails at this less because it seems Sigstore accounts for this, although it requires careful monitoring of the CT Log by the vendors and for the installing machine to be online to check signatures.

(Note that I have learned of Sigstore via this post and it is pretty new to me, but its architecture seems solid at first glance and its claimed industry support from much brighter people than me makes me want to believe their system works.)

I suppose that having a log of the issued signatures and a revocation list would be good to have for
signatures issued by Atlassian for Options 1 and 3 as well.

The options

Regarding option 2: Missing support for airgapped systems makes me not like it as much. Don’t get me wrong, it’s still better than nothing which is what we have today. I’d consider it viable. Note that running this CA might come with a considerable amount of work for Atlassian. I suppose you would integrate it with your Auth0 tenant so that Atlassian accounts registered for a vendor would be able to authenticate with it and get the ability to sign binaries? And you would also do the monitoring of the CT logs and email us when something gets signed with our accounts?

Regarding option 3: As a rule of thumb in cryptographic systems, it’s always better to rely on tools that have been around for some time and have gone through and fixed their share of vulnerabilities. I’m not saying that Bouncy Castle or the cryptographic functions it uses have issues but in practice, cryptographic systems are not usually defeated by defeating the cryptographic primitives (like EdDSA or RSA), but rather the implementations using them. So I would advise against going the homebrew route for this and sticking with jarsigner (Option 1) over Option 3. Option 3 is out for me.

[Edit: I do not mean to imply here that I consider Bouncycastle immature. I mean to say it is relatively low-level and using it correctly can be an issue]

Jarsigner’s CLI output might be hard to parse, but I don’t see many vendors integrating it beyond calling it in a pipeline or manually and failing the pipeline if it returns an error exit code. Besides, it does seem to have a Java API, I think. So personally, I would lean towards option 1.

Another option

I will argue that, as an alternative to the solutions presented here, another option could be similar to option 1 but verifying both signatures from the vendor AND Atlassian. We would be eliminating the single point of compromise, supposing that there is some way of verifying for customers that vendor signatures are actually safe. Maybe something like trust on first use (what ssh does) with vendor certificate signatures being available on their websites for extra security? Or even with known good vendor certificates baked into UPM?

In such a case (or frankly even in option 1), you could maybe even let the admin decide to trust apps that are only signed with the known vendor certificate to facilitate use cases like internal apps or support cases where a vendor is delivering prereleases.

Beyond codesigning

But, and this is my big point, I don’t think Atlassian’s current review processes are sufficient for preventing malware apps from entering the Marketplace. Firstly, there is no review for every version. Secondly, I don’t think the reviews go far enough to be able to detect malware at the moment.

And under any of these models, once an app (version) is in the Marketplace, if it turns out to be malware, it will already have all the necessary signatures to be installed into a customer instance. So to escalate Sysadmin access to code execution, an attacker would just have to install this app.

That is a different threat model for another time, I suppose, but preventing this kind of attack would require an whole different set of controls, such as sandboxing apps with permissions inside the DC instance, which is another really big topic.

Keeping it simple

I’ve been writing for some time, so I can react to posts before mine

As pointed out by @scott.dudley, simply letting Atlassian sign binaries that we upload to the marketplace the current way might just already be a big step forward with close to zero additional effort on the majority of vendor’s parts. I’m going to argue that your 2FA requirement on the marketplace is already a good barrier against a decent amount of attacks.

With this or even just with option 1, if you allowed people to add their own certificates to jarsigner’s keystore, you could account maybe for app-generated code as mentioned by @jcarter as well as the internal app use case mentioned by @steve.behnke. It would also enable the use case of sideloading an app by a vendor who wants to provide a prerelease version of their app for a support case.

I feel like this would be a simple enough solution you could implement without burdening smaller vendors and building up complex systems.

Cheers,
Tobi

[oliver.straesser]

@MalathiVangalapati

• If this feature were to be introduced, is your organization equipped to meet the additional requirements for implementing these changes? What assistance and support would you require to better adopt this feature?

I think we are already very busy with other Atlassian programs, such as Platform7, Cloud Migrations, Access Rule, and Forge Remote, to mention just a few.

But now to important feedback:

To introduce signing is a good idea if it’s not rolled out in a rich and good tested in all settings that happened. with our customers as:

1. Support for OBR packages as well - vendors like we (bitvoodoo), Seibert Media, K15t, Tempo and many more use OBR packages to ship multiple apps and System apps not listed on the Marketplace.

2. Installing an app not listed on the Marketplace has to work instantly, without any flag in UPM or system property. Reason:

   • If the System apps in the OBR packages (see no. 1) aren’t listed on the Marketplace, will they fail to be installed? Please don’t let the solution be, that all System apps need to be listed in the Marketplace and go to the DC approval (they are already part of the main app approval) - this will cause a huge overhead and massive costs on our side (in special DC approval)
   • The usage of custom development shouldn’t bring the system to a state where the Marketplace signing isn’t checked anymore

Otherwise, I assume that most instances will run in such “dev mode,” which can’t be the solution

Cheers
Oli

[scott.dudley]

I still can’t see a substantial change in security by requiring double signatures on JARs: these really just amount to a second “secret” that vendors must prove to Atlassian that they know (on top of account password + 2FA) in order to get Atlassian to apply its master signing signature. I don’t see how customers would be able to check this signature, so it’s really only between the vendor and Atlassian.

From a semantic perspective, requiring the vendor signature then becomes almost equivalent to (but far more complicated than) asking vendors to enter a second “signing password” into MPAC when creating the signature. Yes, this second password could be stolen, but then again, so could the signing secret. There are a few smaller differences between the two, but I don’t actually advocate for either because I don’t think they help substantially.

The goal should be to solve the problem with little or no effort for 99% of vendors, which is automatic single-signing by Atlassian of all JARs uploaded to Marketplace (and also doing whatever magic it takes to make @oliver.straesser ’s OBRs completely signed, top-to-bottom). For those distributing only through the Marketplace, there is no extra effort required. It would also be great if “private” releases uploaded to the Marketplace for a public app could also be signed by Atlassian without forcing it to be a public release, since this is very useful in support contexts.

For the 1% of remaining edge cases (customers with a stable of private plugins, app support team needs to make a private release outside of Marketplace, solutions partners who sell private plugins to their customers, etc), I think there is a use case for vendor signing, but this should be independent and not a condition of Atlassian’s signatures.

The signing tooling should be made available so that people can create their own certificates and sign their own JARs/OBRs. When installing an app, the host product verifies the app against the Atlassian master cert (baked into the host product), or also against a set of arbitrary certificates installed by the admin.

Please let these certs be something that can be added at runtime in a secure manner without recycling the cluster (for example, by scanning for .CRT files dropped somewhere into the product data directory).

This tool should ideally be a simple command-line (and offline) Java program that has one option to generate a cert+key, another to sign an artifact, and another one to validate an artifact and show the signature.

[MalathiVangalapati]

Thank you for all the feedback received thus far. We’ll go through it and start talking about it by March 22nd, as we mentioned in the RFC. Your feedback is really helpful, so please keep it coming.

[PaoloCampanelli]

I wouldn’t reinvent the wheel, we have an extremely common example of an ecosystem in which people can install homebrew/bespoke apps, download them from an official store or from a third-party marketplace: that’s Android, but even macOS notarization is quite similar.

Google’s approach with Android and the Play Store makes sense and highlights some possible limitations and workarounds:

• Custom/in-house apps: either disable signing on your instance, or have a one-off exception which requires higher privileges (“Allow apps from unknown sources”, in Android parlance)
• Apps from the marketplace: Google lets you sign your APKs with a pair of keys that is either owned by you, or managed by the Play Store. The developer decides which one suits their needs:
  • Keys managed by the dev: you can sign everything in your CI/CD and you can have custom versions with a valid signature (think about sending a debugging jar to a customer who has an issue with your app)
  • Keys managed by the marketplace: easier to use for the dev, but gives you less flexibility since you can’t sign apps on your own. For the developer, almost nothing would change compared to now: you upload a jar to the marketplace and Atlassian transparently signs it for you – but you’d lose the option of having custom jars in specific cases.
• Third-party marketplaces (F-droid etc…): you can allowlist one app – or generically, one source – to install other apps, bypassing the signing requirements. I’d expect this to be useful for people who have tons of in-house apps or who have air-gapped instances and have a repository of apps in their LAN

A similar system would guarantee that the most common use cases are covered and leave the choice to the developers and system administrators so that they can pick the best option for their customers.

Personally, I have to say that with the changes to DC apps installation (i.e. you need to set a system property), this doesn’t seem very necessary or urgent.

• If the system property isn’t set, you can only install apps from the Marketplace, which is low risk unless a vendor account is compromised
• If a vendor account is compromised, the only way to be sure that no malicious app is published is if the keys are managed by the vendor, which might be an hassle for small app developers
• If someone’s instance is compromised in a way that the attacker is able to enable the system property for third-party apps, well… it’s too late
• If someone wants to install third-party apps, having them install/allowlist a public key/certificate is one extra step that is annoying for the legit use cases, but probably won’t stop someone from installing JMWE_ScriptRunner_TimeSheets_free_cracked_l33t_h4x0r.DivX.jar

PS: this is my own opinion, it’s not Appfire’s stance on this matter

[paul]

When trying to fix problems we give pre-release versions of our apps to customers to either log more information or verify our change fixes their problem (especially when we can’t reproduce the problem). So whatever you do please allow this to continue in as easy a way as possible. It’s better for the vendors and the customers.

Also, remember that the vendor side of Marketplace is down sometimes so if we are reliant on that for signing then this could affect your customers getting support

[kashev]

I echo everything said previously about the necessity of custom builds to help diagnose customer issues quickly, and add the following - if it becomes necessary to upload custom builds to the marketplace in order to get them signed for customers to install them, the marketplace upload limit per day needs to be dramatically increased. We don’t released multiple public app versions in a day usually, but we often send out multiple custom builds in a day. That, combined with all the breaking Platform 7 changes which will increase the size of almost all app jars given they have to bundle stuff like Guava now, means that this limit will need to be even higher.

[metin]

@scott.dudley

How about a magic solution #4 where vendors upload the .JARs to the Marketplace after authenticating themselves with your preexisting 2FA solution, Atlassian performs whatever magic “blessing” it needs to do, and spits out a link (via UI and REST) to the signed .JAR?

You sign it however you want, you verify it in whatever way suits Atlassian best, and it does not create additional workload for vendors.

How could that work with testing snapshot/nightly builds and internal versions of the apps?

[metin]

May I suggest that you consider an approach based on existing solutions. For example, we include a cryptographic provider, for which Java requires our app to be signed by a trusted certificate issued by Oracle (option 1). If this proven approach is considered good enough for this purpose, why not for Atlassian?

I don’t understand the disadvantage listed as " Parsing jarsigner output may be challenging".

We already use this and it works perfectly well.

[scott.dudley]

@metin

How about a magic solution #4 where vendors upload the .JARs to the Marketplace after authenticating themselves with your preexisting 2FA solution, Atlassian performs whatever magic “blessing” it needs to do, and spits out a link (via UI and REST) to the signed .JAR?

How could that work with testing snapshot/nightly builds and internal versions of the apps?

I assume that either your CI asks Atlassian to sign it (I believe the Marketplace API does not require 2FA when using access tokens), or you configure your test instances to ignore signatures (you’re no worse off than now, right?), or else Atlassian considers an additional system (like I mentioned here) where you are able to configure your specific instances to accept third-party signatures and you self-sign it.

[BrunoMarotta]

• Do you believe that implementing an ‘app signing’ feature for apps would reduce our attack surface and enhance our security posture?
  Yes

• Please provide feedback on the preferred option and explain why you believe it’s the preferred choice.
  Options 1. Then 3

• What specific features or enhancements to this solution would facilitate its implementation and effective management?
  Keep it simple. Please make sure that private builds still work

• If this feature were to be introduced, is your organization equipped to meet the additional requirements for implementing these changes?
  Yes

• What assistance and support would you require to better adopt this feature?
  None for the time being

• Could you outline the current process for ensuring the security of your apps?
  We have fully automated BitBucket/Bamboo pipelines. There are no manual processes.

[metin]

@scott.dudley

[…] or you configure your test instances to ignore signatures (you’re no worse off than now, right?), or else Atlassian considers an additional system (like I mentioned here) where you are able to configure your specific instances to accept third-party signatures and you self-sign it.

I’d rather not have my test instances configured differently. I’d be worried that this could introduce new yet unnecessary complications when hunting down issues.

[scott.dudley]

I’d rather not have my test instances configured differently. I’d be worried that this could introduce new yet unnecessary complications when hunting down issues.

You’d rather not, but it seems like you will have to. I don’t see any indications from the RFC that Atlassian is considering a decentralized model (where JARs are signed only by the vendor, but the vendor key itself is trusted or signed by Atlassian). If they did, this would require the ability to issue revocations, and that adds another large level of complexity (which also does not work well offline).

Atlassian’s presumed goal is to require all code run on stock product instances to be signed directly by Atlassian in order to stop malware actors. In that scenario, they are not going to allow you to directly sign code that can be run anywhere (or else so could the malware actors). That seems to leave you with the option of either agreeing to get your JARs signed by them, configuring your instance to accept signatures from a self-signed certificate (at least if Atlassian likes that idea), or turning off signing entirely.

[metin]

they did, this would require the ability to issue revocations, and that adds another large level of complexity (which also does not work well offline).

The offline problem will be just the same when apps are signed only by Atlassian. Offline means no revocation check available, no matter who signed.

So I don’t see a reason why the proven certificate solution should not work, except that Atlassian might insist to reinvent the wheel (again).

[scott.dudley]

@metin

The offline problem will be just the same when apps are signed only by Atlassian. Offline means no revocation check available, no matter who signed.

I see it as being quite different: if you have a vendor key that is compromised, and a single signature from your vendor key is trusted by all Atlassian products, then you can go on to generate and offline-sign malware1.jar. Atlassian would need a way to revoke your key after discovering malware1.jar is bad, to avoid your also going on to sign malware2.jar and malware3.jar until infinity. Otherwise, they are giving unlimited keys to the kingdom to every vendor, and that defeats the point. Trusting vendor keys would seemingly need to come hand-in-hand with revocation capabilities.

We have to assume that Atlassian’s master signing key will not be compromised and will not ever need to be revoked (and if it does, they have far bigger problems). They can also implement scanning of JARs and whatever other proactive measures they want, and also introduce new measures in real time.

For me, having to install a vendor certificate on a system in order to allow non-Atlassian-signed JARs seems like a reasonable compromise.

So I don’t see a reason why the proven certificate solution should not work, except that Atlassian might insist to reinvent the wheel (again).

I’m personally agnostic about the actual technology used…I am just focusing on who must (or who can optionally) sign things, with the hopes that Atlassian will give us a black box that handles all of the logistics.