Action Required - Atlassian Connect installation lifecycle security improvements

[HanjooSong]

Hi, the SLA date of the AMS ticket will be 31st JAN 2022.
Considering the severity level metrics, original estimate for the due date was to fall on the holiday season. So we have decided to extend it until the end of January as it this is a breaking change.

[HanjooSong]

Hi @LukasGotter as mentioned in other comments,
AMS ticket will be created for you to follow up, and the new remediation due date will be 31st Jan 2022.

[HanjooSong]

Hi, it is recommended to implement backwards compatibility for both lifecycle endpoints. If you are using ACE or ACSB it is already handled by default.
Although it is a rare case, this is to reduce the impact when other underlying services fails to fetch the most recent app version.

There was a recent failure in marketplace place that had impacted some of apps(without backwards compatibility) that was released around this time.

• Marketplace Hub is not creating new app versions

[emre.toptanci]

@HanjooSong , We are using ACSB 2.2.3. We added the “signed-install” flag to true in our descriptor file and did NOT set “allow-symmetric-auth-install-callback” to false. Currently, our service is (and installs are) working fine but we are seeing a lot of messages in the logs about the fallback mechanism being used. We were expecting to see none of these.

We tried setting “allow-symmetric-auth-install-callback” to false and all new installs started getting errors so we removed it. At this point, we don’t have anything else to try.

I fear that when this transition is enforced (whenever that happens), the marketplace will no longer be using the fallback mechanism and our service won’t be able to handle installs.

Can you confirm that you will be following up with AMS tickets, not just the apps that haven’t made this “signed-install” transition but also the ones using the fallback?

[HanjooSong]

Hi @emre.toptanci
Yes, with the recent Marketplace incident, we are aware that it can cause more issues than anticipated. We have an internal ticket to mitigate this issue as much as possible from our end and we won’t be removing the backwards support from ACE or ACSB until we are confident.

Also, I had a look at your case more in depth to identify if you had lost any installation requests (addon: com.obss.plugin.time-in-status).

It seems like due to a recent incident on MPAC, the previous version(1.2.52-AC) may have experienced some delays and upgrade tasks were still running when you have updated your app again on 27th with the new descriptor(1.2.53-AC).

I understand that ideally this should be handled properly from our end, but previously scheduled upgrade task did not acknowledge the latest asymmetric install hook change.
I can assure you that next upgrade task for version 1.2.53-AC was executed as expected.

Thanks!

[KadirFuzaylov]

Hello eveybody,

Found the issue. If you add

"signed-install": "force",

to config.json file, this causes the error on plugin installation. ACE version is 7.4.7

Authentication verification error (401):  Invalid JWT: Algorithm from the header "HS256" does not match

Once I removed it everything is working

[david]

Could you write the current steps out in a vanilla Confluence page which can be a single source of truth?

We could subscribe to the page to find out when changes have occurred.

Then link to it from here. Also update the page with a history of the ACE releases supporting this and what has changed in each release.

Every time someone reads through the 180+ comments on this thread and tries to understand the current state of affairs, a puppy dies

Think about the developers

[AdamMazur]

@HanjooSong

• If I opt-in I will get a new shared secret with every install in Production i.e. installing via MPAC
  → Yes, per-install based shared secret change will be followed within couple of months after you opt-in.
• If I opt-out I will not get a new shared secret with every install in Production i.e. installing via MPAC
  → Yes, it will maintain the existing behaviour. But you may still get a different sharedSecret if MPAC service or any other underlying internal Atlassian service experiences issue that relates to the shared secret. Also, when we need to rotate your app’s secret, it will take up to 24 hours for all Jira instances to be updated with the new one.

Hey, I just want to follow up on this because there are things that are not 100% clear to me.

1. If we opt-in and the app will be updated on the instance will we get a new sharedSecret for each instance and the previous sharedSecret will stop working? I understand that you want to get back to per-installation sharedSecrets, but does it mean that for one instance/tenant we will be getting new sharedSecret on every update OR the sharedSecret received on the first installation (that will be unique per tenant) will never change on updates?

2. You also mentioned that you may need to rotate sharedSecret for an app - how does this procedure look like - how the app is notified about the rotation?

Best regards
Adam

[lsilva]

PSA

In case you have an app written in Node.js and you don’t use Atlassian Connect Express, you might be interested in the Atlassian Connect Auth library.

It supports signed installs and can fallback to the legacy mode during the transition. It’s highly customizable and has types to make it easier if you use TypeScript.

Cheers!

[SamuelVolin]

After upgrading to ACSB 2.2.3 and Spring Boot 2.5.9, and setting “signed-install”: true, we are seeing our logs fill with:

2021-11-04 21:36:38,266 ERROR [http-nio-6800-exec-9] org.apache.juli.logging.DirectJDKLog: Servlet.service() for servlet [dispatcherServlet] threw exception

java.lang.NoClassDefFoundError: org/springframework/boot/web/error/ErrorAttributeOptions

at com.atlassian.connect.spring.internal.AtlassianConnectErrorController.error(AtlassianConnectErrorController.java:44)

errors show when the /installed endpoint is hit. Is not background compatible, currently returns 401.

@HanjooSong was mentioning they were also seeing this error. not sure what else changed.

Edit: ACSB 2.2.3 seems to have mismatched springframework dependencies, but we had success running with 2.2.1. However we now get an error on install:

{"status":401,"error":"Unauthorized","message":"Expecting claim 'qsh' to have value '3256da2a221c8649d5dea341898f83982ea4c951e75f3e560203101ec016e351' but instead it has the value 'context-qsh'","timeStamp":"Fri Nov 05 18:49:10 UTC 2021","trace":null}

okay, so we still have "context-qsh": true set from a previous migration still. is that still required here?

[AdamMazur]

"signed-install": "force" in config.json (in addition to apiMigrations in atlassian-express.json) works for us using ACE 7.4.8.

[AdamMazur]

I created a separate topic for this and got a detailed answer for this there: Per-installation sharedSecret rotations

[AndrewTyson]

I have implemented the fix on ACSB v 2.2.3, Spring Boot Starter v 2.3.1.RELEASE and my application is marked as patched.

However, now I am seeing persistent errors in my AWS ElasticBeanstalk deployment environment:

java.lang.IllegalArgumentException: Collection is empty
java.base/java.util.EnumSet.copyOf(EnumSet.java:179)
org.springframework.boot.web.error.ErrorAttributeOptions.excluding(ErrorAttributeOptions.java:79)
com.atlassian.connect.spring.internal.AtlassianConnectErrorController.error(AtlassianConnectErrorController.java:44) 

Suspect that this is being thrown when the load balancer polls the envionment instances. The error doesn’t affect normal operation of the application but is filling CloudWatch logs with stack trace.

My EB platform is Corretto 11 running on 64bit Amazon Linux.

Has anyone seen the above error, and have any view on how to remediate?

[mateusz.daleki]

@AndrewTyson I suggest to update spring-boot-starter to 2.5.3 version. ACSB v 2.2.3 uses Spring Boot v 2.5.3

[colin.hammond]

@HanjooSong

Update install/uninstall callback authentication to verify the JWT using the public key and with RS256 algorithm.

Can you explain (spell out) what this actually means please. I have got as far as point 3 which returns a public key, now what do i do?

[leonardonunes]

@HanjooSong we upgraded all our apps before 29th October 2021, we are using ace version 7.4.7 and added “signed-install”: true to the atlassian-connect.json file.

All seemed to be working until we noticed that after you change your instance name and URL then the app stops working. Right after you change the instance name the /installed webhook is called, in the payload it sends a new sharedSecret and the new instance url in the baseUrl. The upgraded data gets saved to the addon settings table.

When we try to access the app on the instance that changed it’s name we get the following error:

{} Authentication verification error (400):  Unable to decodeSymmetric JWT token: Error: Signature verification failed for input: eyJ0eXAiOiJKV1QiLJIUzI1NiJ9.eyJzdWIiOiI1...........TMyMjBhYTExYWRjZWYiLCJxc2giOiI1ODJiMTZjMGRjNGExZGQwMDYxNGQyOTB............... with method sha256

[HanjooSong]

Hi @leonardonunes
The bug you’ve mentioned was discovered earlier and has been patched now for Jira. (Confluence will fixed within this week).
However, the change that has lead to this bug was enabled only for the early adoptor’s instances or Ecosystem Beta group instances mainly used for testing. If your customer is still experiencing the same issue and further investigation, please let us know with relevant details.