Multiple clientKeys for the same Jira Cloud instance

Hi,

We’re seeing in our records that some instances have different clientKeys although the baseUrl was always the same.

Reading through the docs I can see the following paragraph:

Identifying key for the Atlassian product instance that the app was installed into. This will never change for a given instance, and is unique across all Atlassian product tenants. This value should be used to key tenant details in your app. The one time the clientKey can change is when a backup taken from a different instance is restored onto the instance. Determining the contract between the instance and app in this situation is tracked by AC-1528 in the Connect backlog.

And in that Jira issue (AC-1528) it says that:

This is because the clientKey is generated from the OAuth consumer key, which is seeded from the instance baseUrl. If the baseUrl changes, already installed add-ons may stop working.

Which, by my understanding, does not apply to this particular issue since the baseUrl did not change.

My question is, therefore, in what scenarios, for the same addon and baseUrl, would we get lifecycle webhooks containing different clientKeys. Also, would it be possible to be notified somehow that clientKey A changed in clientKey B for a particular baseUrl-addon combination?

Thanks.

6 Likes

FYI we have 430 baseUrl/addonKey combinations with multiple clientKeys associated with those records.

We frequently make requests to “all active customers” which means “all customers for whom we think the license is valid” which means “all Jira instances that respond successfully when requesting app data as the add-on user, using the clientkey we have on record” which means that there are Jira instances that have multiple valid clientkeys, somehow. See PSCLOUD-43638 for the giant ball of confusion that this has caused.

Out of curiosity, do you know if these multiple keys are actually valid and it wasn’t a hacker submitting a fake install payload for a real Cloud URL?

Cross-posting from PSCLOUD-43638:

Let me address the multiple clientKey issue which has come up in this discussion. Whenever a Jira instance performs an import, the clientKey for that instance will change. In addition, Connect addons are not imported. So it would not be unusual for a site to be imported and then Connect addons reinstalled following the Import.
In that scenario you will receive multiple install payloads for the site over time, with different clientKeys.
I believe this site has undergone two imports which would explain why you have received three clientKey values.

Please can someone at Atlassian get the docs fixed? This is important information for vendors to know about.

4 Likes

Seems like this is a bit of a mess according to:

2 Likes

Wow! This is really… bad. :frowning:

Especially if you consider that we don’t know when to delete data. In fact this makes the problem worse - we end up with zombie data…

https://community.developer.atlassian.com/t/when-should-i-be-deleting-a-tenants-data

2 Likes

Thanks for illuminating this issue. We’ll have someone take a closer look at this, and per your suggestion @jbevan look at updating the documentation at the very least.

7 Likes

The clientKey is the primary key for identifying a tenant and will not change for the lifetime of the tenant. It is important to note, however, that the site import process effectively creates a new tenant since it may result in the site data being very different from the previous “occupant” of the site. Another consequence of an import is that Connect apps are not included and must be re-installed into the new tenant. Thus, if an app receives a new installation payload with a different clientKey, but for a baseUrl the app has been previously installed into, the app should consider the tenant corresponding to the baseUrl as new and the previous clientKey as obsolete/invalid.

5 Likes

Are the requests from these new tenants with the same baseUrl signed with the previous (or current shared) secret? If they are not signed then it would not be safe to replace a clientKey/baseUrl. New installation payloads are not signed.

We key customer data off the clientKey, so the import process will disconnect the customer from any previous data.

Also we have seen some baseUrls get reused by another customer after going offline for some time.

5 Likes

Hi @jmort,

The app installation payload after the import will not be signed because the new tenant will not have any record of the app being previously installed. You are correct about the import process disconnecting the old tenant from the app since Connect will not notify the app with an appropriate lifecycle event.

Regards,
Dugald

This seems like exactly the kind of scenario that the single shared secret was supposed to address. When will that work be complete?

With the single secret the initial install could be signed too so we could verify that all lifecycle webhooks originated from Atlassian. An alternative would be to use asymmetric signing for installation webhooks so that connect addons can verify that Atlassian generated the webhook.

6 Likes

Regarding the re-install of the connect addon, will an uninstall webhook be sent prior to the import?

Hi @jmort,

Yes, the change to single shared secret will open up possibilities, but currently the initial unsigned installation is the only way we have of distributing the shared secret as there is no app management console for Connect apps. I don’t think there are any dates for when this may be implemented.

If I understand your last question correctly, my understanding is that the import process does not trigger uninstall events even though it does drop all the app installation records. This is a similar problem to AC-863.

Regards,
Dugald

Another issue is that we end up with zombie data that the only way for us to reconcile is to do it manually… This can get even worse if the customer doesn’t talk to us but rather just recreates everything…

5 Likes

When recommendations or requirements like this are asserted as “should” behaviors for apps, it would be really, really great if Atlassian would implement them as features of the official Connect app frameworks, especially when they affect tenant registrations. Working, tested code to act as a reference implementation will be invaluable next to detailed documentation, which as we know frequently falls out of date.

7 Likes

@dmorrow we have the exact same problem, with 323 tenants (URLs) having more than one active clientKey (i.e. a clientKey/OAuth info combination that is recognized by the Jira REST API), with some tenants having more than 2 (the most being 28).

The question is: how can we figure out which one is the current one for a tenant? Is there any REST API that will return the current clientKey for the tenant (Jira Cloud instance)? Or any API that will fail if we use a clientKey that’s not the current one?

Also, since:

  • the clientKey can change for a tenant (Jira instance)
  • the URL of a tenant can change through the “site rename” feature
    what identifier can we safely use to store data associated with a tenant? Not the appEntitlementId, as it’s not available in the /installed webhook.

Thanks,
David

Hi @david2 ,

Based on the description of clientKey field described the reference documentation, if a site import occurs, a new tenant and clientKey is created. Apps installed in the previous tenant will receive a new installation payload and should update the clientKey by looking up existing records by the URL. I realise this is not a great design, but hopefully you can make it work.

If you have existing duplicates, I suggest raising a support ticket at Jira Service Management so we can help de-duplicate these.

Regards,
Dugald

Hi @dmorrow ,

I understand that we’re getting a new installed webhook. However, ACE doesn’t allow accessing the installation records by url so it’s not that easy to update the AddonSettings table. ACE should be handling changing clientKeys but it doesn’t.

As for deleting existing duplicates, are you sure the dev support team will want to look into hundreds of registrations and more than a thousand clientKeys?

In any case, because of the first point, we still need to be able to programmatically figure out which clientKeys are obsolete. Any ideas?

Best
David

2 Likes

Hi @david2 ,

I hadn’t considered ACE. One of us will try to get back to you soon on this. As for the other part of your question, I’m not sure what we can do on our side and how much effort it will take as we don’t have a process for this yet, but we will try to help.

Regards,
Dugald

1 Like