Action Required - Atlassian Connect installation lifecycle security improvements

  • [Update: 12 JUL 2021] Enforcement date is extended from 20th August to 29th October 2021.
  • [Update: 15 JUL 2021] Please upgrade Atlassian Connect Express(ACE) versions to v7.4.0 or later. Previous versions ( v7.1.5 ~ v7.3.x ) automatically opts in to this new feature that results in creating a new app version without the vendor’s action.
    This new ACE upgrade( v7.4.0 ) will not modify your app descriptor which may result in removing "signed-install" field that could have been added unintentionally.
    • When upgrading your app from ( v7.1.5 ~ v7.3.x ) to v7.4.0 , and
      • have previously released a new app version by updating config.json file with "signed-install" field.
        → After upgrading to v7.4.0 the app will work the same as before without additional change, but we highly recommend to migrate signed-install configurations from config.json file to the app descriptor file to prevent any unexpected app version being created automatically in the future.
      • had no intention of opting in to this feature yet.
        → upgrading to v7.4.0 may create another new version as ACE will revert and remove the new signed-install field that was added before.
        If you are happy to stay opt-in and want to prevent MPAC from creating another version automatically, make sure your app descriptor did not change by setting "signed-install" field under "apiMigrations" section.
    • When upgrading from an older version(< v7.1.5 ) of ACE to v7.4.0
      → There won’t be any impact, app developers can choose to opt-in to signed-install feature by updating their descriptor file(Follow the guideline below).

What is changing?

We are making a breaking change to the Connect app lifecycle events to improve the security of the Connect framework. In the install and uninstall callbacks sent by Atlassian to your app, the Authorization JWT token now includes a new signature that you can use to verify that the request is genuine. The signature is generated using an asymmetric keypair using the RS256 algorithm. All Jira and Confluence Connect apps will need to opt-in to this change to receive the updated, signed lifecycle hooks.

Why is it changing?

At installation time, the Connect app server and the host product exchange a shared secret used to create and validate JWT tokens for use in API calls. However, the very first install lifecycle hook is not signed because the secret has not been shared yet to the app server. This makes it difficult to verify that the lifecycle hook was genuinely requested from Atlassian, which may lead Connect apps to be vulnerable to Denial of Service (DoS) and/or Man in the Middle(MitM) attacks and opens the possibility for further exploits when an app secret can be specified by a MitM.

Before

  • Initial install hook is not signed
  • App upgrade install hooks are symmetrically signed with previously shared secret
  • Uninstall hook is also symmetrically signed with the shared secret
  • Initial and upgrade install hooks provide shared secret which is included in its request body
  • Shared secret will be used to create and validate JWT tokens for use in API calls between Atlassian and the app including any subsequent install hooks

After

  • All install(including upgrades)/uninstall hooks will be asymmetrically signed with RS256 (RSA Signature with SHA-256) algorithm
  • Public key will be provided via a CDN to verify the JWT token.
    https://connect-install-keys.atlassian.com/${public_key_id}
  • Initial and upgrade install hooks provide shared secret which is included in its request body
  • Shared secret will be used to create and validate JWT tokens for use in API calls between Atlassian and the app, excluding install and uninstall hooks

Is my app impacted?

If your app uses an authentication descriptor setting other than none , then your app is impacted and will need to be updated.

This change affects apps regardless of how they are installed; apps listed on the Marketplace and apps installed via the UPM “dev mode” are affected .

Even if your app skips validating install hooks with any reasons, we still recommend to opt-in to the breaking changes and make the install lifecycle hook secure as soon as possible.

Apps running on Bitbucket Connect are not affected .

What do I need to do?

Running an Atlassian-supported Connect framework

  1. Update Atlassian Connect Spring Boot, or Atlassian Connect Express to the latest version †
    • Atlassian Connect Express: v7.4.0 or later
    • Atlassian Connect Spring Boot: v2.2.0 or later
  2. Check if your app has enabled the updated asymmetric install hook by inspecting signed-install field in your app descriptor. Also make sure that your app baseUrl is properly configured. ‡
  3. If you need more time to comply with the change, Opt-out by setting signed-install element to false. (See how this can be configured for ACE/ACSB from below).
  4. Test that your app is correctly authenticating install hooks from a test instance and deploy your app

† ACE framework will opt-in to the new behaviour(Asymmetric install hook authentication) by default, but for apps that uses ACSB framework will have to update atlassian-connect.json to enrol to the new feature.

‡ Your descriptor should define new element like below:

# 1. Atlassian Connect Express
# After updating the ACE version to 'v7.4.0' or later, "signed-install" attribute 
# must be added in the descriptor file. 
{
  ...
  "baseUrl": "{{localBaseUrl}}",
  "apiMigrations": {
    "gdpr": true,
    "signed-install": true
  }
  ...
}

# In your `config.json`
# Check that 'localBaseUrl' in your config.json matches the baseUrl from the descriptor
# The app baseUrl will be used to verify the audience claim.
{
  ...
  "signed-install": "force", 
  "product": "jira",
  "production": {
    "localBaseUrl": "https://app.base.url",  # This is used to verify 'audience' claim
    ...
  }
}

# Additionally, you can choose to disable the legacy install hook by setting 
# "signed-install" to "force" in "config.json" as global variable. 

# NOTE: Setting this to "enable", "disable" will still be used
# to be compatible with previous unstable ACE versions (v7.1.5 ~ v7.3.x) 
# but for v7.4.0 and later, please do not use this field unless you want to set it to "force".
# 2. Atlassian Connect Spring Boot
# After updating the ACSB version to 'v2.2.0' or higer, "signed-install" attribute 
# must be added in the descriptor file. 
# (newly created apps will have this added by default)
{
  ...
  "baseUrl": "http://app.base.url",  # This is used to verify 'audience' claim
  "apiMigrations": {
     "signed-install": true
  }
  ...
}
# Additionally, you can choose to disable the legacy install hook by setting 
atlassian.connect.fallback-install-hook-enabled=false`

Running a custom implementation

If you’re using a framework other than Atlassian Connect Spring Boot, or Atlassian Connect Express, then please follow these steps to ensure your app correctly verifies install lifecycle hooks.

  1. Opt-in to the updated asymmetric install hook by adding a new signed-install element to the apiMigrations section of the app descriptor.†
  2. Extract JWT token from install hook request’s Authorization header.
  3. Decode only the header of the JWT token and extract kid (How to decode a JWT token header)
  4. Use the extracted kid to fetch the corresponding public key from CDN.‡
  5. Verify the JWT token using the public key and retrieve the decoded JWT body to perform post-installation steps after verifying the claims including the audience( appBaseUrl ).

† Your descriptor should look like this:

...
  "baseUrl": "http://app.base.url",  # This is used to verify 'audience' claim
  "apiMigrations": {
     "signed-install": true
  }
...

‡ CDN url should have key id as the path to fetch the public key

https://connect-install-keys.atlassian.com/${kid}

Keep in mind that this public key gets rotated regularly (More than once a day).

For reference, a pseudo-code example of JWT verification that uses RSA public key from Atlassian CDN:

// Authenticate install/uninstall lifecycle hook
Function AuthenticateAsymmetricJWT(request)
  // Get Authorization header from request: `Authorization: JWT ${jwt_token}`
  jwt_token <- request.Header['Authorization'] or request.QueryString['jwt']
  
  // Decode 
  jwt_header <- DecodeProtectedHeader(jwt_token)
  
  // Get key id from JWT header
  key_id <- jwt_header.kid
  if (key_id is empty) 
    return Error(Unauthorized)

  // Fetch RSA Public key string(PEM format)
  rsa_public_key <- fetch(`https://connect-install-keys.atlassian.com/${key_id}`);
  if (rsa_public_key is empty) 
    return Error(Unauthorized)

  // Verify signature and decode jwt_body and jwt_header
  // Verifying the `aud` claim (app baseUrl in your descriptor file) is important. 
  // Also make sure that the external lib validates other required claims such as `exp`   
  expectedAudience = "https://your.app.baseUrl";
  expectedIssuer = "host_client_key";
  { jwt_body } <- external.jwtlib.JWTVerify(jwt_token, rsa_public_key, expectedAudience, expectedIssuer);
  if (jwt_body is empty || caught exception during verify) 
    return Error(Unauthorized)
  else 
    return jwt_body;

// Decoding jwt header from the token: Use external lib if possible
Function DecodeProtectedHeader(token)
  [encoded_header, encoded_body, encoded_signature] <- token.split(".")
  header <- base64.decode(encoded_header)
  return header

By when do I need to do it?

Enforcement of this breaking change is planned for all apps by 29 Oct 2021. If you do not patch your app to mitigate this vulnerability and opt-in to the signed-install API migration before this date, your app may not be able to install / update.

Additional minor patch will be required for the apps using ACE / ACSB framework to cleanup support for legacy install callbacks using shared secret.

12 Likes

Hey @HanjooSong, could you please clarify how this

and this

can be valid at the same time wrt. uninstall events? I suppose this is an oversight and the last sentence should actually read 'excluding install and uninstall hooks` - right?

Btw. thanks for this announcement and the change! I think this improves both security and development workflow (e.g. when switching between app environments such as dev or staging in a confluence instance) at the same time.

2 Likes

You do realize that for a lot of folks that you’re asking for a new development effort with no notice during vacation months? So 2 months is really something like 2-4 weeks.

8 Likes

Ok.

Let me start by saying that I’m really happy with the fact that install & uninstall event payloads will now be signed. Hooray team :tada:

I’m just having a lot of trouble wrapping my head around the architectural design choice for creating Yet Another Point Of Possible Failure. I’m especially baffled by this part:

You are basically telling all Connect app developers that are not using ACE to add multiple lines of code (add caching layer to infrastructure, decode token, fetch public key from cache, verify token, if token fails try and fetch new public key from CDN, verify token, cache new public key). This should also be done each and every install / uninstall request.

The ones that are using ACE / ACSB can only hope that you made this as resilient as possible.

Can you please tell me why you decided to not just add a “Shared Secrets” section to MPAC app management pages, in which vendors can generate (& rotate) a single shared secrets for their app which they can put in their environment variables which will be used for every install & uninstall event specifically to their app key?

This is an acceptable best practice approach and would have saved everyone multiple lines of code (incl. the requirement of adding a caching layer to their app).

12 Likes

Why would the key rotation prevent caching as long as the key ID is specified in the JWT header? Give us a TTL for the key so we can cache them and only need to load them if the cache doesn’t know an ID.

If you remove a public key then you probably won’t sign any requests with the corresponding private key anymore, right? If you remove the private key then we won’t see any further requests signed with that key :slight_smile:

If a private key is leaked then you need to reach out to us anyway. Otherwise we could never store the last fetched key to cover CDN outages.

6 Likes

Isn’t this change in collision with the timeouts?

  • I think ACSB cancels the /install transaction if it takes more than 3 seconds,
  • But Atlassian servers don’t necessarily cancel it, so the plugin might be installed,
  • But the point is that timeouts are very very very tight,
  • If you add a request to the CDN to fetch the public key, and accept that a timeout from the CDN allows us to fallback on the cached public key,
  • Then the spec is non-deterministic. For some customers, apps will install successfully, for others it will say “Installed!” but the app will have reverted the transaction, etc.

We only save ONE record in the ACSB install hook, and yet we often see ACSB cancelling our transaction because of timeouts. Wouldn’t it be better to discuss your implementation proposal with the community before directly starting an SLA deadline against us? At least this time your implementation is published, which is nice.

  • Can you please review the maximum timeouts for each HTTP call, and ensure that requests can be successful if we are close to the maximum time?
  • Can you ensure that it is possible for a customer to reinstall the app, even if the host believe the app was already installed, and/or even if the ACSB believes the app is installed but not the server?
4 Likes

Hi @HanjooSong Can you provide more clarity on this? Is this additional minor patch required before August 20?

And is ACE really tested? The last releases had a couple of bugs.

@HanjooSong Follow up question: do we need both:

"apiMigrations": {
     "signed-install": true
  }

and

"signed-install": "enable",

And can you provide a minimal app descriptor with the required fields?

  1. What happens when an app’s baseurl changes? Will The installation call have the baseurl that Jira “thinks” it’s calling? (Or is the expectation for us to have in our code every base url we’ve historically used? (Edit: I’m guessing that a new base url will mean a new KID to be generated - so if anyone is planning on caching things locally - it’s not a single KID per app).

  2. Is the public key secured or signed somehow? Ie. How do I know that the key I’m getting is the “real” key?

  3. What’s the monitoring and escalation path for this new service? If we reject a customer’s upgrade - what do they see? Will we lose the upgrade?

1 Like

I agree that the comment around CDN raises confusion and not really useful.
I have updated it to simply state that the key pair will be rotated frequently.

2 Likes

If you are using ACE framework, adding "signed-install": "enable", to your config.json file is enough.

For ACSB, you will have to update the descriptor to include

"apiMigrations": {
     "signed-install": true
  }

Hi, for the first 2 questions.

  1. What happens when an app’s baseurl changes?

Audience claim of the JWT token will be the baseUrl defined in the descriptor of the app’s installed version, which is also part of the requestUrl from Jira’s install hook request. Also, kid is not tied to the baseUrl.

Is the public key secured or signed somehow?

Public key is signed with a matching private key using RS256 algorithm. This private key is used to sign the JWT signature which app can verify with the public key that we provide. If the JWT signature is different the publicKey is not a correct one.

1 Like

Is this additional minor patch required before August 20?
The patch will remove support of the old install/uninstall callback authentication. It will not be a breaking change for the apps that have already opted in to the new feature. This will not be enforced but encouraged for security benefit.
And please let us know with any bugs that you’ve encountered, and I will try to unblock you as soon as possible.

1 Like

asking for a new development effort with no notice during vacation months?

It is true that the timeline is a bit tight for the changes related to vulnerabilities.
Apps can still choose to opt-out even after 20 Aug 2021 and we will closely communicate with those app vendors before removing support for the old behaviour.
The “breaking change” on this date will most likely be for the apps does not respond to this change at all (signed-install field not defined in the app descriptor)

2 Likes

Hi @HanjooSong!

  1. Will the public key be rotated?
  2. How can we validate the public key? Since its retrieval also vulnerable to MitM attack.

Hi @anton2

  1. Will the public key be rotated?

Yes, it will be rotated more than 1 times a day, and it is not a periodic scheduled job.

  1. How can we validate the public key?

As its name suggests, Public Key will not matter how you have retrieved it.
If you were able to

  1. verify the JWT signature which is signed with a matching privateKey from Atlassian,
  2. and check if the audience claim of the JWT body matches the app baseUrl,
    it means that the request was properly made from us.
1 Like

I don’t think that’s entirely true. If an attacker is able to modify our DNS resolver / your DNS entries they could fake the CDN and make us download their public key rather than yours. In that case they obviously can also control the corresponding private key and create and sign any JWT as they like.

However it’s questionable what such an attack could be used for, given that they still wouldn’t be able to get hold of the shared secret used for app<->Atlassian Cloud communication.
In the worst case they could sort of DoS our app because we now have an incorrect tenant record in our DBs. Or use it as a first step for further attacks against app APIs that don’t go through the host application (like REST APIs).

Therefore I think we should be very careful that public keys are properly downloaded from the Atlassian CDN and not from somewhere else. The origin of a public key is only irrelevant if there’s another way to validate it.

The question is if it enough to trust DNS. In browsers this is solved by using a well known key (root / intermediate certificates) to sign website public keys (=the website certificate). In theory something similar could be done here as well: Use a long lived key pair to sign the public keys and apps could hardcode the public key of that pair.

Not sure if that’s a bit too much though. I suppose customers concerned about their data will answer this with ‘NO’ :slightly_smiling_face:

2 Likes

Hanjoo is on leave so I’ll just quickly jump in here to say we 100% agree with this and it is actually how it is done today. The public key is served over TLS and the response includes a valid certificate chain.

The client should always verify the public key is served from the https://connect-install-keys.atlassian.com domain and the response is signed with a valid certificate signed by a trusted root authority. This should happen automatically provided your application server is configured in the correct way.

It is also possible to do certificate pinning on application servers, though it’s less common most likely because DNS spoofing is a lot harder on servers than it’s in browsers (where it’s commonly done via a compromised wifi). However if you have concerns, I can ask one of our security people to jump in on the discussion.

3 Likes

@HanjooSong
I am trying to implement this and I got this body in the installed hook

{
        "key": "my-app-key",
        "clientKey": "252c289c-ebc6-3cf7-959d-9620395e3e37",
        "oauthClientId": "xxx==",
        "publicKey": "MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCF/QdxiV3VXMpyW2QTKhEhibh6EwOLPX0/vnds3ymMWp3shH3x/ANyYksjXxYX8REVYL6HwW5efB/TkY3OxfZvAx0y5uPTctov9gw358PIX13NIFso2Y1n/JZpZVt+K9QqMPDIGDj8bFbCMLL5eTQo0nYqAhN6HVTVubt6eWT8EQIDAQAB",
        "sharedSecret": "xxx",
        "serverVersion": "100166",
        "pluginsVersion": "1001.0.0-SNAPSHOT",
        "baseUrl": "https://jira.atlassian.net",
        "productType": "jira",
        "description": "Atlassian JIRA at https://jira.atlassian.net ",
        "eventType": "installed"
    }

Where do we get the public key id from?

P.S. I tried to assume that the public key is the public key id - but I get 404 from CDN.

Hi @moveworkforward ,

The keyId is not in the install hook, it’s in the header of the JWT token. See

The linked page includes some examples on how to do it in Java, what you need to do depends on which language or library you’re using.

Let us know if you have any problems with this, most JWT libraries should support extracting the key id.