RFC-24: Data Center Grey API Removal

General Request for Comments (RFC)
, , , , , , ,
1

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building a shared understanding of a topic. It expresses a technical solution but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines : keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

  • Publish: 11 September 2023
  • Discuss: 22 September 2023
  • Resolve: 11 October 2023

Summary

Dear Marketplace Partners,

At Atlassian Data Center, we plan to modify the range of third-party party libraries available for the Marketplace apps.

Our objective is to achieve compliance with increasingly challenging security vulnerability resolution time standards. At the same time, we want to ensure that Marketplace apps remain unaffected by the library upgrades we need to perform to patch these vulnerabilities.

To achieve this, we plan to coordinate a Platform release across DC products in Q1/Q2 2024. We will provide technical details, such as the list of exported libraries, around Oct/Nov 2023.

With this RFC, we aim to engage with you early to explain the problem and gather feedback on the solution.

Problem

Currently, many third-party libraries are exposed to the ecosystem (aka Grey API). Any major upgrade to these libraries risks introducing changes that can break Marketplace apps that are dependent on them. In order to mitigate that risk, we at Atlassian Data Center have often held off from upgrading libraries. In the case of security issues, we have often had to create forks of no longer supported libraries and apply the patches to these manually.

This led us to have multiple libraries forked and out-of-date.

At the same time:

  • In the last year, we observed increased demand from customers to quickly upgrade third-party libraries that carry any security issues (even those that are of low severity);
  • Customers don’t trust the security of the forked libraries and require costly pen-test reports, which are not scalable for Atlassian;
  • The fact that we are using an outdated dependency poses a risk that we may not be able to address security issues within a reasonable timeframe without adversely impacting our customers.

Solution

We need to improve our dependencies and processes to the point where we can confidently upgrade and release any dependency immediately while making sure the apps stay stable.

To achieve this we want to significantly reduce the set of third-party libraries available to the Marketplace apps. With this, we would declare the remaining list of libraries as an API. This aims to prevent any future breaking changes for Marketplace apps while Atlassian upgrades the third-party libraries. Marketplace partners will need to define these dependencies individually and will be responsible for upgrades in all non-public libraries at their own pace while adhering to the Atlassian Marketplace Security program requirements.

To introduce this change, we will coordinate a Platform release across DC products in Q1/Q2 2024 with a regular process of EAPs shipped before.

As the next step, Atlassian aims to continuously replace unsupported third-party libraries and get to the point of proactively upgrading libraries to the latest version. The proposed solution will allow the Marketplace apps to follow that path at their own pace.

Timeline

  • Q4 2023 → Share technical project details, including a list of exported libraries
  • Q4 2023/Q1 2024 → Ship the Early Access Preview (EAP) versions of all products
  • Q1/Q2 2024 → Ship the new Platform version in a coordinated DC products release

This timeline is subject to change.

Actions

We acknowledge that Marketplace partners whose apps rely on the removed third-party libraries will need to start managing these dependencies themselves, including patching vulnerabilities.

We will monitor and ensure Marketplace apps security compliance through the AMS program.

We also understand that technical project details are required for you to fully understand and react to the impact of the proposed change.

At this stage, we want to understand better:

  • Are any hard blockers preventing you from independently managing the dependencies (for example, JAR size limits)?
  • How can Atlassian support you in this transition?

As always, please leave your feedback in the comments below. We will collect it by the end of September. At this time, we will also try to answer all the questions.

5 Likes
2

Thanks for the early heads-up.

As you already mentioned it will be hard to point out concrete problems without any further details.

Just some early thoughts:

  • Upload size limits on AMKT are already now a problem for us, forcing us to distribute uploading new app versions over multiple days / user accounts when releasing multiple apps at a time. This should certainly be increased.
  • The host app docs should also consider this change where heap size settings are recommended.
  • The next announcement should also consider shared libraries that apps can’t easily bundle, for example spring or logging libraries.
9 Likes
3

Thanks for the early heads up in this is much appreciated.

For me I don’t see any blockers at the moment, but if libraries will be required to be packaged by the app, then please make sure you expect larger app libraries and make sure that the publishing of new apps in not impacted by this change.

4

Will this RFC remain in discussion until the published list of impacted dependencies is available?

The impact to apps and vendors can only be determined and discussed once that has been done.

2 Likes
5

Is this about the Java or Frontend 3rd Party Libs? Or maybe both?

Something I noticed about the log4j vulnerability that caused a lot of work for all customers:

  • Even after pathing the jira system, many customers couldn’t say if one of the installed plugins is still affected and needs to be patched
  • As a result they checked the changelog for every installed plugin or created a support ticket

Depending on how many libs are exported and provided by atlassian, this change may increase the customer workload in case a vulnerability like log4j happens again.
The reason for that is that quite a lot (not sure about that) of plugins may have used the 3rd Party Lib from atlassian and a fix within atlassian would also fix it for many installed plugins.
There is however the downside of updates with changed behaviour / backwards incompatible changes. I’m in conflict with myself about this one, but shouldn’t we test new jira versions either way?

I’m not sure if something like this would be possible, but wouldn’t it be great if the customer could do a vulnerability check live on their instance taking into account the installed plugins? (using the information from the pom file)

6

I believe this is primarily about OSGi exported dependencies.

One piece of feedback we provided was that having a bill of materials for apps may be beneficial, or the ability to see an apps dependencies from the UPM. In practice this would probably be quite tricky.

1 Like
7

My main concerns are around Log4j and Lucene. ScriptRunner dips into the Log4j API directly to dynamically register logging appenders against Jira/Confluence logging, this cannot be done via SLF4J.

Confluence provides the com.atlassian.confluence.logging.sink.LogConsumerService public API which was introduced with the Log4j2 upgrade. If API access over OSGi is to be removed, an equivalent in Jira would be necessary.

ScriptRunner also depends on various Lucene packages/classes due to our custom JQL indexing code.

For convenience, I’ve attached a list of all OSGi imports for ScriptRunner for Jira.

sr4j-osgi-imports.txt (31.3 KB)

7 Likes
8

I assume that any library that’s used in anything exposed by jira-api will continue to be exported as transient dependencies.
However, certain apps, in particular scripting apps, access more than just jira-api. They also access jira-core classes. In that case, we might end up accessing a class or member that uses a class exported by a 3rd party library, and if we can’t access that library directly and have to import the same library ourselves (not with “provided” scope), we’ll end up with a different instance of the class and the app will break. That’s the biggest risk of having multiple copies of the same classes loaded in the same JVM.

And more generally, I agree with @m.herrmann: while there’s a risk of breaking other apps when Atlassian upgrades 3rd-party libraries in Jira, it will never break customer instances since app vendors need to explicitly declare app compatibility with every new version of Jira. On the other hand, having a single copy of libraries means they will always be up to date in Jira instances, without the need for every app to be upgraded as well.

1 Like
9

Hi @rlander,

I’m not sure if you were implying the ability to see dynamic dependencies created at runtime, but at least the static dependencies are already visible on a hidden page if you punch in the URL: /plugins/servlet/upm/osgi

2 Likes
10

The community would be helped by clarity about which types of APIs are in-scope of this RFC, as well as to understand what exactly is going to happen to those APIs that are in-scope.

I am making the assumption that, to spell out the meaning of “reduce the set of third-party libraries available”, what you mean is that these packages will no longer be exported to plugins via OSGi. Is that correct?

I also strongly hope that this applies truly only to third-party packages, and that vendors can be given some assurances that this policy will not also be applied to Atlassian-originating packages. (If this RFC is instead supposed to be about preventing apps from using anything except the features in jira-api and confluence-api, then we need to know now so that we can provide targeted commentary.)

For example, I hope that vendors will still be able to safely import any existing com.atlassian.* package that is currently exposed. Those are not, after all, third-party packages, so they are out-of-scope…right? Similar for other Atlassian-acquired packages (com.pyxis.greenhopper.* and so on).

I also hope that there will also be some room for negotiation with this list, and that certain third-party libraries will be left in that are widely used by apps. For Confluence, this would include Struts and Velocity, for example (and these already seemingly have major upgrades announced long in advance via EAPs).

I think it would also be the most helpful to list all of the packages that you are removing from the OSGi exports, instead of (or perhaps in addition to) listing the ones that you are keeping exported.

7 Likes
11

Another thought to counter the size-explosion a bit:

If Atlassian (or the ecosystem) has some experience with with tree-shaking of Java libs it might be a good idea to set up a guide or even some support in the SDK / AMPS for it.

Ideally the JVM wouldn’t need to load dozens of full copies of commons-lang3 etc. just because everyone is using StringUtils :slight_smile:

6 Likes
12

Thank you for the feedback. We will look into this.

13

Thank you for the feedback. We will consider increasing the upload size limits.

14

Thank you for this RFC. I echo @scott.dudley’s wish for a more detailed clarification of which packages are in scope of this change. We would like a detailed list with the diff in osgi exports with this change introduced.

How will Atlassian handle the upgrade path in this case, and compatibility? I am particularly worried about these scenarios:

  1. The customer updates Jira, and everything breaks down since they didn’t update the plugins simultaneously. Customer creates support ticket to every app vendor as well as Atlassian, and need to jump through hoops to get stuff working again.
  2. A customer running an older version of Jira (which doesn’s have this change implemented) clicks update or otherwise gets their hands on an app artifact which is built for “the new way”, and everything is likely to break down. Support ticket is created, and lots of troubleshooting for customer and ourselves.

From the moment in Q2 when version X of the platform is launched, we as a vendor would from what I understand, need to build different artifacts of our new app versions to maintain compatibility with older versions of the host product while also being compatible with this new architecture. We want to offer updates to our apps also for customers who take their time and are running on an LTS version of the host product. How will we make the distinction between these different artifacts regarding compatibility? With today’s UPM architecture, I don’t see how to make a distinction between new app versions built for “new way” and “old way”. I don’t think it’s enough to set compatibility for every “app version for Data Center”, but another piece of information is needed to identify the compatibility of a given artifact available on the Marketplace.

With today’s mechanisms, we might imagine a workaround where for example every odd-numbered patch version is compatible with old stuff, and every even-numbered version is compatible with the new way, and make sure that the compatibility range for these every-other versions is flicked accordingly for each build. However this approach sounds messy, and we would instead welcome a way to expose “old generation build” and “new generation build” artifacts of the same app version, with granular control of compatibility.

1 Like
15

Thank you for the feedback.

Is this about the Java or Frontend 3rd Party Libs?

  • It is primarily about Java

…but shouldn’t we test new Jira versions either way?

  • we would still test it of course, but the risk of introducing breaking changes would be significantly reduced

I’m not sure if something like this would be possible, but wouldn’t it be great if the customer could do a vulnerability check live on their instance taking into account the installed plugins? (using the information from the pom file)

  • customers are already using their own vulnerability scanners to understand the security posture of Atlassian products and Marketplace apps. We plan to release SBOM for Atlassian DC products to make it easier for them.
1 Like
16

Thank you for the feedback. Once we have a complete list of exported libraries we will be open to discussing the potential impact on various plugins and possible remediations.

17

We plan to collect feedback on this particular RFC by the 22nd of September, but we will reopen the discussion once we share more technical details around Oct/Nov.

18

Thank you for the feedback.

what you mean is that these packages will no longer be exported to plugins via OSGi. Is that correct?

Yes, that’s correct.

I also strongly hope that this applies truly only to third-party packages, and that vendors can be given some assurances that this policy will not also be applied to Atlassian-originating packages.

We will be also reducing the scope of our own packages. However, our main goal is to keep the functionality needed by the plugins while simplifying the API. We will be analyzing our current exports and reviewing what is helpful and what can be hidden. For sure we want to limit the exposure of our implementations.

I also hope that there will also be some room for negotiation with this list and that certain third-party libraries will be left in that are widely used by apps…

If a library provides vital functionality that cannot be replaced we won’t be removing it. We are taking into account the usage of the API on the plugin’s side and for sure we will include the feedback about certain packages.

19

This is extremely concerning and needs more clarification. For example could I interpret this as packages from jira-core no longer being exported, as they are not classed as public API?

There is a huge amount of necessary dependency on non sanctioned APIs.

How does restricting non third party dependency packages have any relation to improving the security of the platform? This will only hurt customers.

8 Likes
20

As Reece said, please don’t conflate these two completely separate concerns.

Also don’t kid yourself that you will know “what is helpful”. See also: https://www.hyrumslaw.com/.

9 Likes