Upcoming changes to Data Center App Approval

kbrazulewicz · November 30, 2021, 10:13am

At Atlassian, one of our goals is to make sure customers can trust our software to be safe and secure. That is why every Data Center and server product at Atlassian is subject to a third party dependencies security scan for known vulnerabilities. This way we are able to detect potential issues early enough to reduce any risks for our customers.

Over the course of years we’ve seen the evolution of mechanisms used in Atlassian Data Center Engineering to discover security vulnerabilities. There are security scanners, Bug Bounty Program, and more. Customers rely on Atlassian software and trust us with their data every day.

Atlassian Data Center products are part of a thriving Ecosystem and the vast majority of deployments host multiple Data Center apps. The security of such a complex setup depends on the security of each of its components.

We want to share our experience in this area and help partners align with Atlassian Security Standards.

What is changing?

We are extending the Data Center App Approval process and introducing a third party security scanner for all Data Center apps.

Starting from February 2022, we’ll require all apps submitted for Data Center App Approval to be free from critical- and high-severity security vulnerabilities in third party dependencies (Bamboo DC Apps Program will follow these requirements once it’s launched). Learn more about the technical aspects of the security scan process and find out how to run it in your development environment.

Starting from April 2022, we’ll track vulnerabilities in the Atlassian Marketplace Security Jira project. Atlassian Marketplace Security is our one-stop-shop for vulnerability management, where partners can go to review all their vulnerabilities, statuses, due dates, sources, and severities. All issues in AMS describe the vulnerability in detail, and all questions about an issue can be addressed in the comments of the issue. Learn more about AMS, and review our Security Bug Fix Policy for Marketplace Apps.

Why is it changing?

It is critical to address vulnerabilities as soon as possible to ensure our customers can continuously trust the Atlassian Ecosystem.

What do I need to do?

If the annual review cycle of your app is scheduled for February or March 2022, the Data Center apps review team will contact you through a DCHELP ticket in the upcoming weeks and provide a list of vulnerabilities the scanner has detected. We will also reach out to partners who own apps with the highest number of detected vulnerabilities.

We want to ensure that we create enough room for you to discuss and act on the security scanner results.

Last but not least, please ensure that the security contact in your partner profile are up to date and that you follow Vulnerability review practices for Atlassian marketplace partners. It is important to complete those steps before the rollout of the security scanner integration with the Atlassian Marketplace Security Jira project.

Thank you for taking this seriously. Please review the technical aspect of dependency scanning and consider including it into the CI/CD pipeline of your Data Center app.

Update (Dec 8th)

Thank you very much for all the feedback on this post. The sheer size of it indicates how important this process is for you. We’ve identified two major concerns in this feedback:

The tooling that Atlassian suggested is insufficient as it provides different scan results than tooling which Atlassian uses internally to validate apps.
Approval Process requirements after Feb 2022 need to be rephrased to clarify the expectations and address non-obvious scenarios.

We are working to address those points, and we will provide an update by December 17th. Please stay tuned!

Update (Dec 17th)

For the last week, we have been testing alternative scanner solutions and discussing legal aspects of those approaches. We are not yet satisfied with the outcomes thus we will continue the investigation in the tooling area.
At the same time, we are drafting updates to the Approval Process documentation published on developers.atlassian.com.
Given the upcoming holiday period, we will provide another update by January 14th.

Update (Jan 14th)

Thank you very much for your patience. For the last few weeks, we were validating available scanner solutions and came up with updates to the Approval Process documentation.
We clarified that during the Approval Process, you need to provide a scan report free of critical- and high-severity vulnerabilities in the libraries bundled with your app. You can also choose a Software Composition Analysis tool of your choice to create this report.
For more details please see Security Scanner for DC Apps documentation.
We’ll be monitoring this post to answer your questions.

emre.toptanci · November 30, 2021, 12:00pm

Security vulnerabilities usually need significant efforts (and quite often architectural changes) to fix so seeing those vulnerabilities at the last minute (as part of an initial DC certification or annual renewal) might cause too much rework on our side.

Will it be possible for marketplace vendors to submit an app for a scan so we can see the results at an earlier stage?

danielwester · November 30, 2021, 12:10pm

For my own clarification “third party dependencies”. Can that please be documented to as to what that means? From the perspective of this process - what is third party?

Ie if library X is brought in due to a dependency on a confluence maven module - is that third party? If so what’s the escalation path with Atlassian?

Thanks.

/Daniel

jens · November 30, 2021, 2:11pm

I would expect the scan to cover all libraries bundled in the OBR files. There’s no point in scanning other libraries provided by the host apps (in the scope of this scan) because app vendors can’t update them.

… I guess we could bundle newer versions, but then we’d end up with a lot of duplicated libraries in the JVM.

But I’d also be interested in more details. For example: will you scan only Java libraries referenced in pom.xml files or do you also scan for frontend libraries.
This would be interesting for us so we can tune the scanners in our CI to find the same problems before Atlassian finds them

aragot · November 30, 2021, 3:24pm

On the public page, it says:

At the moment only maven and gradle package managers are supported from the box.

Just to confirm, maven, gradle and npm are supported, since snyk itself is deployed using npm. Is it correct? Is esbuild supported too?

OleksandrMetelytsia · November 30, 2021, 4:49pm

Hi @emre.toptanci. This particular scanner at the moment is able to identify vulnerabilities in the third-party dependencies, so in most cases, there is a straightforward way to fix - upgrade the dependency version.
We encourage app partners to embed third-party dependency scanning into the CI, but also anyone could ask in the DCHELP ticket to scan the app from the Atlassian side to get scan results at an earlier stage.

OleksandrMetelytsia · November 30, 2021, 4:52pm

Hi @danielwester. Third-party means not provide Atlassian dependencies. So if in your example vulnerability is in the Atlassian module, then an internal ticket would be raised for the confluence dev team to fix, but not against your app.

OleksandrMetelytsia · November 30, 2021, 4:55pm

hi @jens . Yes, exactly, the scanner is not going to scan provided dependencies.
There is no scanning of frontend libraries yet, but we are working in these directions.
And it’s awesome that you already have CI to scan for dependency vulnerabilities

markrekveld · November 30, 2021, 7:37pm

Is there also a way to scan the app artifact (obr, jar) it self?

The reason I ask is that I have worked with people over at Snyk to get accurate reporting of the Jenkins Integration for Jira app, but Snyk currently doesn’t correctly report on dependencies for the app. For some reason the dependency management overrides are not taken into account, resulting is a very large list of false positives making the scanner utterly useless.

aragot · November 30, 2021, 10:22pm

Yes, exactly, the scanner is not going to scan provided dependencies.

I disagree, it seems to report all vulnerabilities in provided dependencies, including for example H2, which is a “provided” dependency of Confluence, which we don’t import directly. Here are the commands mentionned on dc-apps-security-scanner:

snyk test --all-projects --detection-depth=10 --json-file-output=results.json
# -> Returns all vulns of "provided" dependencies such as:
#   ✗ Deserialization of Untrusted Data [Critical Severity][https://snyk.io/vuln/SNYK-JAVA-COMTHOUGHTWORKSXSTREAM-1088331] in com.thoughtworks.xstream:xstream@1.1.1
#   introduced by com.atlassian.confluence:confluence@7.4.0 > com.thoughtworks.xstream:xstream@1.1.1

As said, we’ve only included Confluence as “provided”. The other command on the page:

mvn dependency:tree -DoutputType=dot -DoutputFile=maven_dependency_tree.gv
# The file returns: "com.atlassian.confluence:confluence:jar:7.4.0:provided" -> com.h2database:h2:jar:1.4.196:provided" ;

Can we simply use Maven’s OWASP security scanner, which you don’t mention on your page? Will we get the same results? Here is a sample configuration:

<build>
    <plugins>
        <plugin>
          <groupId>org.owasp</groupId>
          <artifactId>dependency-check-maven</artifactId>
          <version>6.3.2</version>
          <configuration>
              <!-- Even if it doesn't fail, it produces a report -->
              <failBuildOnCVSS>7</failBuildOnCVSS>
              <skipProvidedScope>true</skipProvidedScope>
              <skipRuntimeScope>true</skipRuntimeScope>
              <suppressionFile>src/main/other/owasp-vulnerability-suppression.xml</suppressionFile>
              ...

scottohara · November 30, 2021, 10:49pm

A fresh, empty Confluence plugin skeleton created with atlas-create-confluence-plugin (using version 8.2.7 of the SDK) includes:

<dependency>
  <groupId>com.google.code.gson</groupId>
  <artifactId>gson</artifactId>
  <version>2.2.2-atlassian-1</version>
</dependency>

As I’m sure you’re aware, this is currently flagged by Synk as a High vulnerability:

Question 1: Is this the sort of thing that is expected to be ignored by the scanner?

Question 2: (this may not be the appropriate thread to ask, but…) This dependency appears in the generated pom.xml under the comment “WIRED TEST RUNNER DEPENDENCIES”. If my app doesn’t use the wired test runner, can I safely remove this dependency?

OleksandrMetelytsia · December 1, 2021, 1:17pm

Hi @aragot. If .jar file has a package.json file inside, then it will be scanned. Does esbuild use package.json?

OleksandrMetelytsia · December 1, 2021, 1:22pm

@aragot. Yes you could use Maven’s OWASP security scanner. I could not guarantee that Snyk and Maven’s OWASP security scanner will return exactly the same list of findings, but it should much in most cases.

Snyk does not have an option to skip provided dependencies, but Security Scanner has this feature. So we are not going to create any tickets for provided dependencies.

OleksandrMetelytsia · December 1, 2021, 1:24pm

Yes, please remove all unused dependencies from your pom.xml.

AndriiStashyshyn · December 1, 2021, 1:30pm

hi @markrekveld. Security Scanner has a feature of skipping provided dependencies, so the resulting list should be correct. You could ask in your DCHELP ticket to scan your app and check if the scan results look correct.

markrekveld · December 1, 2021, 1:55pm

Hi @AndriiStashyshyn,

For me scanning provided dependencies is not the issue.
The issue is that Snyk is incorrectly identifying dependencies.
For instance in a previous scan it identified a vuln on a Spring dependency, and I was not even using that version of the dependency, the project was using a newer version but for some reason the scanner didn’t take this into account. So I don’t have any trust at this time in Snyk as they have not been able to correctly scan my any of projects that use a more complex dependency management build structure.

Currently I’m looking into X-Ray of JFrog, so far so good (at least when looking at the reported dependency versions used by the project).
Does the new scanner requirement also allow using X-Ray?

rlander · December 1, 2021, 1:58pm

My understanding is that the security scanner Atlassian will use to scan app jars behaves differently to the instructions provided for running Snyk against the actual source code repository of the app.

This seems to be leading to confusion, as running Snyk in the way described reports many false positives for provided dependencies which will not end up in the built jar.

What is the rationale for not making the security scanner the DC approval team are using public? Ideally I would like to be able to run a scan in CI and see identical results to those that apparently can be requested via a DCHELP ticket.

danielwester · December 1, 2021, 2:07pm

I would echo this. Can we please get hold of the scanner that Atlassian will be using?

Right now snyk reports multiple critical vulnerabilities due to me relying on Jira’s api (provided in maven) - ie:

  <dependency>
            <groupId>com.atlassian.jira</groupId>
            <artifactId>jira-api</artifactId>
            <version>${jira.version}</version>
            <scope>provided</scope>
        </dependency>

for 8.6.0 and upwards (including 8.20.0) have issues.

I’m assuming that your internal scanner does not have this issue. Can we either get it either documented or get hold of proper tooling that will allow us to see what Atlassian will see?

rlander · December 1, 2021, 2:08pm

fwiw I have run Snyk with the instructions provided and the resulting scan is full of hundreds of false positives for runtime dependencies outside our control, the amount of noise makes the scan useless to us.

AndriiStashyshyn · December 1, 2021, 2:32pm

Snyk scanning uses the output from command: maven dependency:tree. You can validate your dependency tree with that command.
And Yes, you can use JFrog Xray.