RFC-39 Allowing installation of only signed apps on customer instances

General Request for Comments (RFC)
, , ,
21

Thanks for sharing this as RFC with us vendors. :clap:

  • Do you believe that implementing an ‘app signing’ feature for apps would reduce our attack surface and enhance our security posture?
    • Only partially. Especially if not fully embedded in a bigger security and threats model (revocation etc.) and key management “reimplemented” by every vendor. See how google provide it as a service for the Android Play store: Use Play App Signing - Play Console Help
  • Please provide feedback on the preferred option and explain why you believe it’s the preferred choice.
    • Preferred Option 1 with app signing service from Atlassian.
  • What specific features or enhancements to this solution would facilitate its implementation and effective management?
    • Signing as a service by Atlassian (incl. vendor signing!)
  • If this feature were to be introduced, is your organization equipped to meet the additional requirements for implementing these changes? What assistance and support would you require to better adopt this feature?
    • See above, this should be required as little as vendor work as possible.
  • Could you outline the current process for ensuring the security of your apps?
  • Feel free to share any additional comments or thoughts on this feature.
    • Keep it simple, learn from existing solutions for similar problem spaces: see google app store.
    • Support for beta/bugfix releases for customers to try out by vendors only
    • Support for apps in early development not yet in marketplace
    • We echo many points others already mentioned in this thread!
  • If you are interested in a follow-up 1:1 with our team to share or know more, please leave a comment on this post and we will get back
    • No need at this moment from our perspective.
2 Likes
22

Thank you for the RFC.

Do you believe that implementing an ‘app signing’ feature for apps would reduce our attack surface and enhance our security posture?

Yes it’s partly true, but it will introduce more complexity to the ecosystem.

Please provide feedback on the preferred option and explain why you believe it’s the preferred choice.

First Option-1, and then Option-2. With a signing service provided by Atlassian.
Since we’re dealing multiple apps, we’re aiming for a solution that requires minimal changes.

What specific features or enhancements to this solution would facilitate its implementation and effective management?

  • Current apps should keep working seamlessly
  • No hassle for the instances behind the firewall
  • Easy custom build distribution (sometimes we give a specific customer a custom build to resolve an issue)
  • Well designed procedure for the compromised vendor keys

Thanks,
Tolga

23

For me the biggest gain I think would be in further securing vendor marketplace accounts. For example marketplace having a means to validate the signature of an uploaded jar, using keys from the vendor which can not be modified in marketplace itself.

Right now if our marketplace account gets compromised, an attacker is a couple of clicks away from distributing malware to a significant portion of DC instances. We try to protect these accounts, but security is never absolute, recent attacks on the Atlassian products by nation state actors (e.g the CloudFlare hack) should give some pause.

Signing does not inherently make the platform more secure. I think it would be trivial for a bad actor to get a malicious app onto marketplace and signed. How much static analysis of apps does the marketplace really do? It’d be an arms race, and I doubt Atlassian will come out as the winner in this.

Vendors could also be either directly compromised or coopted into bundling backdoors into their apps by bad actors. For all we know there are already backdoors in apps on marketplace. That’s ignoring the legitimate apps that explicitly offer unrestricted access to the underlying system, which this solution also won’t fix.

If an instance that is internet connected gets compromised, marketplace will be a means to get further access to the system, regardless of whether app signing is in place, due to what apps can currently do on DC.

With all that in consideration, this feels like a box ticking exercise that doesn’t really improve security posture. Under that lens I would opt for Atlassian to sign apps with their keys at time of upload transparently, hide the complexity from vendors.

In addition, the ability for vendors to create custom builds of their apps and sign them is a hard requirement, we all need to do this from time to time. Ideally this would be done without having to do something silly like creating a private version on MPAC.

Do we really need double signing from the vendor and Atlassian? As a large vendor we can definitely do it, but it is yet another burden on smaller vendors.

4 Likes
24

Hello everyone,

First of all thanks a lot @MalathiVangalapati for launching this RFC.

I will vote for the Option-1 or the Option-3, in this order. The reason is that Option-2 may be a little bit to complex so it will require more development time for partners and I do not see the extra value we will get is worth it.

I also think that this will not all the bad actors risks but it is a good initiative if the development required is no so complex as some of us may have several Data Center apps that may be updated.

Moreover, we should take into account the apps that are not in the Marketplace so there may be a lot of customers with custom apps for their own use cases. It seems this will not affect them I would mention it only to have an eye here.

Have a nice day!

25

Hi All,

Thanks for the feedback and suggestions regarding the RFC. Just to emphasize, app signing is one of the steps we’re considering, along with other capabilities like Multi-factor authentication and allowing Web Sudo operations that are limited by IP addresses, as part of our efforts to enhance our security stance. App signing serves several key purposes:

  1. Verifying Plugin Integrity: It ensures that both custom and marketplace plugins haven’t been tampered with since they were signed, maintaining their integrity.
  2. Establishing Trust and Authenticity: By confirming that a plugin originates from a genuine source rather than a malicious entity, app signing boosts trust and authenticity.
  3. Protection Against Attacks: It assists in safeguarding against different attack vectors, such as uploading local JARs via API/UI in UPM or through the file system, and marketplace installations via UPM.

Our proposed solution

The solution we are leaning on is option 3. This option gives us more confidence in controlling the signing process while adhering to regulations and policies. We also considered option 1, which is similar to the Android APK signature schema v1. This schema has shown some limitations in terms of performance and security. This is because only entries that were part of the archive during the signing are taken into account, which leaves some blind spots in terms of attack surface.
Signing the entire archive allows the signing process to be performant and secure. This approach has been adopted since the APK signature schemas v2. However, unlike Android, we decided not to embed the signature into the app bundle. This process would require some byte manipulations and make the double-signing process complex.

In terms of impact, option 3 allows us to gradually phase this project and implement the “Marketplace Trust zone” independently. This should already help reduce some risks with a lesser impact on app vendors. Only Atlassian and customers having custom apps are required to sign apps. In this initial phase, Atlassian should automatically sign reviewed apps and their versions. Customers will be able to trust any public key or certificates stored in a dedicated folder. This should allow them to sign and install private or custom apps.

Also note, that already installed apps will not be required to be signed in. However they need to be signed in case of any upgrades (new versions installation).

Addressing some recurring themes

We’re taking an iterative approach here, gradually addressing the issues at hand. We’re exploring various aspects of the app signing like Atlassian app signing service, mitigating supply chain attacks, among other concerns. Some of these aspects are potential candidates for future iterations.

A few recurring themes have surfaced from our discussions:

  1. Handling Custom Plugins: It’s crucial that customers retain the ability to install custom or private plugins. We’re considering options such as signing these plugins and locally trusting their key/certificate. During the initial phases, we’ll maintain support for downgrading through the same process. Support for custom apps is essential, and customers will be able to trust any certificates dropped in a dedicated folder.

  2. Exploring Holistic Solutions: What we’ve outlined in this RFC represents the initial stage of app signing. We anticipate its evolution over time. For example, future iterations could involve verifying signatures from both the vendor and Atlassian. This approach would mitigate single points of compromise, provided there’s a way to ensure the safety of vendor signatures for customers.

This project is a long-term endeavor, requiring several iterations for a progressive rollout and ensuring backward compatibility.

  1. App signing service: Regarding the suggestion of having an “App signing service” by Atlassian, it’s indeed a promising idea. Such a service could alleviate the burden of key/certificate management for app vendors and enable Atlassian to enforce robust security policies regarding key rotation and revocation. This could be considered once app vendors are mandated to sign their apps.

  2. Enhancing Marketplace app review process : Agreed, the current marketplace review process doesn’t catch malicious code. However, we conduct regular checks on existing Marketplace apps to ensure they don’t contain security vulnerabilities. We’ve forwarded this feedback to our Marketplace team and will keep you updated on our progress.

4 Likes
26

@MalathiVangalapati the other key theme here: Vendors need to be able to sign app binaries that are not public on marketplace.

This cannot require the customer having to install a custom certificate. Many Atlassian administrators do not have access to the filesystem of the machine hosting the application. This is also a huge burden on our customer support team, who would then have to guide a customer through setting this all up, likely during an active incident.

Debug builds of apps are commonly used to support customers, usually with additional logging baked in. We need to be able to sign these and allow customers to upload them, without any additional work for the customer.

This is another reason why removing the upload option from the UPM prematurely harms customers, they can no longer easily install these custom debug builds.

9 Likes
28

Thanks for patience on this RFC. @MalathiVangalapati and team are still working towards a resolution. We’ve changed the resolution date from 10 Apr, which we already missed, to end of this week, 19 Apr.

1 Like
30

Hi all,

Thank you for your valuable feedback. It helped us clarify the constraints and use cases that need to be addressed for app signing. I am pleased to confirm that we have selected solution 3 for app signing. This approach allows Atlassian to fully control the signing and verification process, enabling us to harden the signing algorithms and ensure compliance with various regulations. Internally, Bouncy Castle (BC) will be used for generating app signatures and during the verification process.

Phased rollout

Another significant reason for opting for the BC approach is the ability to have a phased rollout. For the initial phase, Atlassian Marketplace will sign validated apps. The signature will be then verified during the installation phase using Atlassian public certificate. In this phase, the impact is mainly on the Marketplace.

Custom apps

Customers having custom apps will need to sign them as well. Their public keys certificates should be pinned in the filesystem in a specific directory to have UPM pick them up. Those keys will be trusted and enable our customers to enhance their security practices by allowing only trusted apps to be installed.

Private builds

During the initial phase, app vendors will have the possibility to generate valid certificates. These certificates will be trusted by UPM during installation and will allow app vendors to share private builds directly with our customers.

Dev and CI settings

UPM will be updated to offer the ability to disable app signing verification for local development and CI. This will be implemented using dedicated feature flags, decoupled from the “dev mode”

Grace period

This project will significantly impact the installation process. A grace period will be allowed where customers will still be able to install unsigned plugins. However, they will receive a clear warning regarding this action and be informed about the end date of the grace period.