Unable to decode Symmetric JWT token error after database dump restored on another server

Hello,

We are experiencing a strange behavior of the Jira Cloud app using Atlassian Connect.

What we did:

  1. installed an app using the private listing into our test Jira Cloud.
  2. created a database dump with tenant info of that test Jira.
  3. restored the dump in another database instance located on another server
  4. adjusted database connection string so that our Cloud Jira app used the new database.

Since nothing changed in the app descriptor our assumption was that the app would just work, but we received the following error when opening the app:

“Unable to decode Symmetric JWT token. Signature verification failed for input:…”

During one of our tests we noticed that sharedSecret in AddonSettings.val column changed after the app was restarted, but was not reinstalled in the test Jira.

After we reinstalled the app in test Jira sharedSecret also changed and the app started working normally.

This issue currently blocks us from migrating to a new infrastructure because in this case, customers will have to reinstall the app which is not how it should be.

Any suggestions are appreciated.

Thanks,
Vasiliy

2 Likes

Hi Vasiliy,
I’d like to understand the steps you have taken. Please confirm my understanding:

installed an app using the private listing into our test Jira Cloud.

This is normal Connect app installation process

created a database dump with tenant info of that test Jira.
restored the dump in another database instance located on another server
adjusted database connection string so that our Cloud Jira app used the new database.

To be clear, this is a database dump of your app’s data and these steps do not involve any operation on the Jira side. Your service is still located at the same URL and is just connected to the new database.

Since Connect code on the Jira side has no idea about your database location this should indeed work as expected.

Is there any window where the service was online with the old database after the database dump was made? Is it possible that the app’s secret was updated between steps 2 and 3 so that the restored DB is missing a change?

I’d suggest you create a DEVHELP ticket with details on your Jira instance name and timing of your various operations. We can then look at our logs to see what is happening on our side.

Hi @cmacneill

Thank you for your feedback, I’ve created a DEVHELP ticket - Jira Service Management

The time between steps 2 and 3 took about 5 minutes so technically the secret could change. I’ve provided more details in the support ticket.

Regards,
Vasiliy

Hi @vasiliy.krokha i believe we have encountered a similar issue. When we create a new app version on the marketplace for jira all existing customers using the app will nolonger be able to use the app unless they perform the app update. once they update the app to the new version everything works fine.

image

we have raised this issue with Atlassian and hoping they give us a solution because now we are not able to create new versions since it impacts customers.

did you have any solution based on your ticket?

Check the last value in the AddonSettings id sequence

SELECT last_value FROM AddonSettings_id_seq;

most likely its overwriting existing rows

Hi @FadiHawli

Unfortunately no. As it blocked our “go live”, we accepted the risk that customers would need to reinstall the app, but hopefully it went smoothly.

thanks @vasiliy.krokha our developers have been able to reproduce this issue in our development environments by changing the sharedsecret value for the jira instance installation of our app. This value is created when a customer installs our app for the first time.

It seems when we create a new version of the app Jira is updating the sharedsecret from the client side and it no longer matches the record we have for the customers jira instance and hence a JWT token error occurs.

This is not the first time we have created new versions of our app. The last time was 6 months ago and we had this problem. We contacted Atlassian support and they said they couldn’t reproduce this issue. We ignored this thinking this was a one time issue but unfortunately last week we released our latest version and again the problem occurred. After debugging this we noticed that the only way to reproduce the exact same issue is to change the sharedsecret in our database and the issue was noticed.

At this stage we have escalated the problem to Atlassian support with the extra information on how to reproduce this issue and hopefully we get an answer.

For now we have handled this error in our app by providing a friendly message to the customer that a new version was available and to update the app. Once they update our app works.

We cannot continue like this as we intend to introduce a new app version every 6 months for major features we introduce.

thanks

1 Like

@FadiHawli - check my response, you need to check the sequence in your database. I had the same issue and the issue is not even fixed when you fix the sequence. You then need to remove rows that are affected by the ID duplication and have customers re-install the app.

1 Like

Thanks @ernst.stefan appreciate the support.

we solved this problem by moving our custom code outside the /installed life cycle event
image
and using the recommended /enabled event.
image

since doing this jira is receiving the 200 ok success on installation/upgrades for our app and working fine.