Action Required - Atlassian Connect installation lifecycle security improvements

[james.dellow]

UPDATE 2: I deployed to production and ran into this again, so I reverted back to using jwt.SymmetricAlgorithm.HS256 again to fix the issue.

UPDATE: Don’t know why, but this is working now.

@HanjooSong can you please check out this related issue with atlassian-jwt discussed in this thread:

Atlassian-jwt 2.0 Algorithm from the header “HS256” does not match - Jira Development / Jira Cloud - The Atlassian Developer Community

This caused a lot of wasted time, but subject to testing, I think I resolved it.

I had to use decoded = jwt.decodeSymmetric(token, secret, jwt.SymmetricAlgorithm.HS256); yet decoded = jwt.decodeAsymmetric(token, response, ‘RS256’); works.

[scottohara]

We are trying to determine what actions (if any) we need to take here, and we’re having some trouble interpreting the original post and many of the subsequent comments, as some appear to be contradictory .

Background info:

• We have a number of Connect apps (for both JIRA and Confluence) that use ACE
• Versions of ACE differ across these apps (some are currently using v7.1.4, others are using the latest v7.3.0.
• It is our understanding that the apps on v7.1.4 will need to be upgraded to use a more recent version of ACE.
• In all apps, our atlassian-connect.json descriptors have the following values:

{
  ...
  "lifecycle": {
    "installed": "/installed"
  },
  "authentication": {
    "type": "jwt"
  },
  "apiMigrations": {
    "gdpr": true,
    "context-qsh": true
  }
  ...
}

• We don’t (currently) have the signed-install key set in any of our config.json files

Questions:

Our confusion is around the following statements (emphasis is ours):

@HanjooSong

ACE framework will opt-in to the new behaviour(Asymmetric install hook authentication) by default

@HanjooSong

If you are using ACE framework, adding "signed-install": "enable", to your config.json file is enough.

@HanjooSong

The “breaking change” on this date will most likely be for the apps does not respond to this change at all ( signed-install field not defined in the app descriptor)

Q: If ACE opts-in by default, do we still need to explicitly set "signed-install": "enabled" in our config.json file? And where you say “not defined in the app descriptor” you mean config.json, right?

@HanjooSong

Under the recently announced Vulnerability Management Program for Marketplace Apps , Atlassian will create AMS issues for eligible apps with a target SLA to patch this vulnerability.

We have not yet seen any AMS tickets assigned to us for our apps in relation to this issue (despite the fact that we know some apps require action on our part, e.g. the ones currently on ACE v7.1.4).

Q: Should we have been notified of this issue via AMS tickets yet?

(We are Silver tier partners, if that makes any difference).

Closing thoughts:

It is very concerning to us that there has been zero communication of this breaking change, other than here on CDAC. Had we not stumbled on this post, we would not have been aware of the change, despite the deadline of 20 Aug 2021 having been announced over 3 weeks ago, and with only 4-5 weeks left.

While we appreciate that CDAC is a primary communication channel, in the case of a breaking change such as this, we would have expected something a bit more explicit (such as the email that was sent regarding the context-qsh vulnerability back in April 2021).

In our opinion this whole change has been very poorly communicated, both in the wording/language used in the above post, and the mechanism by which it is being communicated to vendors.

[HanjooSong]

Hi @scottohara

do we still need to explicitly set "signed-install": "enabled" in our config.json file

Updating ACE version to v7.1.5 or higher will opt-in to this feature by default. So it is redundant to set "signed-install": "enabled" in your config.json.

And by app descriptor I am referring to the atlassian-connect.json file(which is controlled from config.json for ACE framework).

We are planning to extend the deadline of this change and make sure it is communicated to all vendors via email. I will have this page updated to remove confusion around the deadline in the mean time.

[HanjooSong]

@dciupureanu

Do you think we can get the Connect Inspector tool updated to use the new signed install feature? This would help a lot with testing.

https://connect-inspector.services.atlassian.com/page/start

Hi, Connect Inspector tool has been updated now. Thanks.

[marc]

@HanjooSong Is my understanding correct: Versions of ACE after 7.1.5 modify the atlassian-connect.json that is served from our app. ACE adds automatically

apiMigrations: {
  "signed-install": true
}

I was surprised by this, as it seems not documented. We have atlassian-connect.json in source control to know what the capabilities of our app were at which point in time. I’d argue that changing atlassian-connect.json is a major new thing in ACE. This change also automatically released a new version of our app on marketplace, which also was a surprise, as we did not change atlassian-connect.json.

Are there any other changes done to atlassian-connect.json when serving through ACE?

[danielwester]

For apps that have authentication none in the descriptor - do they need to do anything?

  "authentication": {
    "type": "none"
  }

Or is this a case of where we should just add the signed-install thing just in case?

[HanjooSong]

Hi,
You do not need to do anything for authentication type none as we will ignore them when we collect reports before the enforcement date.
FYI, new version of ACE framework will automatically add "signed-install":true to the descriptor but will have no effect.

[HanjooSong]

Hi @marc ,
We updated ACE to add signed install support and it was set to automatically opt-in apps to this new feature. We realise now this was not the correct approach and a subsequent ACE update removes this opt-in. Our apologies if you app unexpectedly registered a new version due to this change.

[HanjooSong]

Hi @scottohara
Apologies for the confusion with ACE and its default opt-in feature.
We agree that modifying app descriptor from the framework was not a good idea, and have removed the default opt-in with a recent release.
The settings from config.json will still work as before with the new v7.4.0, it is recommend to edit the app descriptor file directly and add "signed-install":true under apiMigrations to opt-in.
Please ask any questions if you have.

[HanjooSong]

Hi @ErkkiLepre , We have added support for multiple baseUrl for verifying the audience claims with the recent v7.4.0 release. You can define list of allowed app base urls in you config.json as below.

{
  ...
  "production": {
    "allowedBaseUrls" : [
       "https://other-domain.herokuapp.com"
     ],
    ...
  }
}

[RhysDiab]

Hi @HanjooSong

I recently upgraded my ACE version 7.3.0 and added "signed-install": "enable" to my config.js.

Not long after I noticed that webhooks started to become “patchy”.

Sometimes POST request would be sent when an issue was updated in Jira, sometimes it wouldn’t be. There would be stretches where no event would be emitted at all. Sometimes the event would be emitted a few minutes later with a retry header, sometimes not. But the original POST to my API was never sent.

Likewise, sometimes app installation worked, other times it returned a 503 error.

But when I go to my app logs, the /installed endpoint is never called.

1. Could these issues be related to ACE v7.3.0?

2. Is there any chance you could check if this new version alters webhook behaviour? I’m not sure if the issue is related to the ACE version but It’s the only change I’ve made to my app recently and am trying to narrow it down.

3. Would you suggest updating immediately to v7.4.0 and adding "signed-install": true to my descriptor as a potential fix?

Thanks in advance.

Rhys

[HanjooSong]

Hi @RhysDiab ,

1. Could these issues be related to ACE v7.3.0 ?
   There hasn’t been any changes related to the webhook in ACE, also 503 status code is not what ACE returns explicitly from the framework as far as I know. I believe it may not be related to the upgrade.

2. Is there any chance you could check if this new version alters webhook behaviour?
   We did not notice any issues around webhook with recent ACE versions, but I will keep on investigating around this area and keep you posted.

3. Would you suggest updating immediately to v7.4.0 and adding "signed-install": true to my descriptor as a potential fix?
   The minor patch from v7.3.0 to v7.4.0 only changes the default opt-in behaviour. Patching your app will not mitigate the issue with the webhook if ACE did introduce the bug.

Additionally, if this issue keeps blocking you from releasing your app, creating DEVHELP ticket with more detail around your app and test instance will help with the investigation.
Thank you.

[RhysDiab]

Thanks @HanjooSong

My app is already in production.

Can you post a link to the DEVHELP portal?

I can’t find it anywhere in the UI.

UPDATE: found the support portal here

Cheers,

Rhys

[ErkkiLepre]

@HanjooSong

addon.authenticateInstall() was intended for custom install/uninstall callback authentication.

Hi @HanjooSong ,

When we add addon.authenticateInstall() to our uninstall route, the uninstall authentication fails for installations that were done before the change. What’s the best way to identify the new asymetrically signed requests? Or is there another way to work around this?

And another question: for non-customised apps, what’s the best way to confirm, that the installed / uninstalled routes are being authenticated properly?

[HanjooSong]

Hi @ErkkiLepre
For non-customised apps, the framework will fallback to use old method of authentication if the JWT header alg is not RS256. Basically the force flag is introduced to disable this fallback and fully gain the security benefit.
But I understand the problem that you are having with custom uninstalls, and I think it should be something addon.authenticateInstall() should support out of the box, just as the default framework behaviour. Thanks again for reporting this issue, I will have this bug sorted as soon as possible.

[HanjooSong]

Hi @ErkkiLepre please find latest ACE (v7.4.1) for the addon.authenticateInstall() middleware patch.

[ErkkiLepre]

Thanks for the prompt response, but unfortunately the new version doesn’t seem to help.

To reiterate, the scenario is, that the customer has installed the app using an older version of the app. We update to ACE 7.4.1 and add addon.authenticateInstall() to the /uninstalled route. The the uninstallation fails with the error 'Authentication verification error (401): Invalid JWT: Algorithm from the header "HS256" does not match'

[HanjooSong]

Hi @ErkkiLepre , sorry to hear that the patch did not work as expected. I am assuming it could be the “force” behaviour in your config.json file.
Can you try with removing that setting if it is there, and have “singed-install” configuration only from the descriptor files?

{
  ...
  "baseUrl": "{{localBaseUrl}}",
  "apiMigrations": {
    "gdpr": true,
    "signed-install": true
  }
  ...
}

[ErkkiLepre]

Hi @HanjooSong ,

As a matter of fact, we have two apps - one that serves both Jira and Confluence, and one traditional one that serves only Confluence, and I was experimenting with the latter one yesterday. Nonetheless, moving the configuration from config.json to the descriptor solved the issue. Thanks.

One more question. authenticateInstall overwrites req.context (originally set in request.js in ACE). What is the purpose of this change? We rely on req.context in our installation flow, and this causes changes for us. We can work around this, no problems there. Just trying to understand the purpose of this change and whether we should rely on req.context to begin with.

[rohan]

How do we validate enabled and disabled events? Will a JWT be included for those as well?