Connect uninstalled webhook received when upgrading to Forge version (same app key) - Is this Expected?

Forge Adopting Forge from Connect
, , ,
1

We maintain a Connect app that we migrated to Forge module by module. Until today we had two types of installations:

  1. Connect-on-Forge version (new installations and upgrades from Connect)

  2. Legacy Connect version (customers that never upgraded to our Connect-on-Forge version);

Our latest major version removed the last Connect artefacts from the manifest (lifecycle, scopes, etc). We kept app.connect.key, as the docs require.

Before:

app:
  connect:
    key: our-connect-key
    remote: connect
    authentication: jwt
connectModules:
  jira:lifecycle:
    - key: lifecycle-events
      installed: /lifecycle/install
      uninstalled: /lifecycle/uninstall

After:

app:
  connect:
    key: our-connect-key
modules:
  preUninstall:
    - key: app-uninstall-trigger
      endpoint: app-uninstall-endpoint

No connectModules, no connect.remote, no connect.authentication. The new version also drops the read:connect-jira and write:connect-jira scopes.

What we are observing on upgrades to the latest major version

Sites already on the Connect-on-Forge version → The upgrade sends only avi:forge:upgraded:app. We receive no Connect callback of any kind, even though the previous version declared jira:lifecycle with an uninstalled URL. This is the behaviour we hoped for.

Sites still on the legacy Connect version → We receive the Connect uninstalled lifecycle POST first, then avi:forge:installed:app (not avi:forge:upgraded:app).

Note 1: It’s possible that the uninstall + install combination is result of manual actions, but the fact that we are observing it consistently, and only on the Connect to Forge path, makes it suspicious.

Note 2: Past upgrades from the legacy Connect version to the Connect-on-Forge version didn’t trigger an uninstall.

Problems

  1. The uninstalled handler on our remote is a destructive operation (meanwhile, we release a temporary fix).

  2. Between Connect uninstalled and Forge avi:forge:installed:app, the app becomes silent (i.e. no Jira events sent to our remote).

Questions

  1. Is the behaviour we are observing expected when upgrading a legacy Connect version to a fully adopted Forge version (same app key)?

  2. For a site coming from the legacy Connect version, is avi:forge:installed:app the expected event, rather than avi:forge:upgraded:app?

  3. Assuming we are observing the correct behaviour, is there any signal in the callback payload or the Forge event that lets an app tell a migration apart from a genuine uninstall?

Thanks for any clarification on this topic!

2

Hey @JPires, thanks for bringing this to our attention!

No, I have raised https://ecosystem.atlassian.net/jira/browse/FRGE-2276 to track it.

I would expect that Connect → Pure Forge:

  1. Forge avi:forge:installed:app event only

Yes, this is the behaviour you should expect. I acknowledge this is a bit confusing, but full disclosure: Connect and Forge are managed by different systems internally. The first “upgrade” of a Connect installation to a Forge installation (with or without connectModules) is the first sighting of the Forge app by the system that emits the avi:forge:installed:app event. Contrast this to Connect-on-Forge, where you’ve previously deployed the Forge manifest containing the connectModule, and therefore any subsequent changes are emitted as avi:forge:upgraded:app.

Connect → Connect-on-Forge:

  1. Connect /installed hook with both Connect clientKey and Forge installationId per Retrieving the Connect clientKey in Forge
  2. Forge avi:forge:installed:app

Connect-on-Forge → Forge

  1. Forge avi:forge:upgraded:app

Unfortunately there’s nothing in either the Connect uninstalled payload or the Forge installation event that distinguishes a migration from a genuine uninstall.

The one place that signal does exist is the Connect installed callback you receive while you still have a Connect lifecycle module in your Forge manifest - that payload includes the Forge installationId, so if your app passes through a Connect-on-Forge step you can record the mapping at that point. This isn’t available to you now that you’ve moved to a pure Forge manifest.

The recommended approach is to recover the Connect clientKey from your Forge install trigger using the app properties API, as described here: Retrieving the Connect clientKey in Forge. That lets you detect pre-existing Connect data for the installation from your Forge installation handler and treat it as a migration. This should also allow you to programmatically recover any of the installationId → clientKey mappings you lost as a result of the uninstalled hook.

Thanks again for raising.

3

Hey @MitchMcCue , thanks for your quick reply!

We are already able to link the Forge install to the original Connect installation, so our only problem was the Connect uninstalled being fired, which resulted in the teardown of the integration on our side with loss of customer data.

Our quick fix was to defer the uninstall and deal with it later. For now we have a manual process to identify and teardown genuine Connect uninstalls (i.e. Connect uninstalls that are not followed by a Forge install). We are working on the automation of this process to remove operational burden from the team. Are you able to give us an estimate of when the bug will be fixed? That will inform us about the need to invest in the automated process.

Another question, can you confirm if the following assumptions are correct when upgrading directly from Connect to Forge?

  1. Connect app is uninstalled
    a. Connect webhooks are no longer triggered
    b. API calls using Connect auth fail (unhauthorized)
  2. Forge app is installed
  • Between (1) and (2) the application effectively stops working (no outbound events fired; no inbound API calls accepted).

We are observing a gap of close to 1 min. between (1) and (2), but we have an outlier that reached 12 min., which is a bit concerning.

Thanks!

4

Hey @JPires, I’m testing the change now, should hopefully hit production early next week at the latest.

I’ll track down someone from the Forge side of the fence to come answer your question RE events and auth, standby.

5

Hey @JPires, the change has fully rolled out in Confluence + Jira.

As for the radio silence between the Connect uninstallation and Forge installation, the Forge team would appreciate an ECOHELP ticket with trace IDs that they can dig into to provide a confident answer.

You can raise one here: Jira Service Management

Cheers,

Mitch

6

Hey @MitchMcCue , thanks!

I’ve found no new occurrences in the past 24h. I’ll keep monitoring and let you know if something comes up, but looking good so far.

Regarding the “radio silence” topic, I’d be happy with a statement of the expected behaviour to avoid going through loops if what we’re observing is expected :slight_smile: . I’ll open the ticket nevertheless.

Thanks!

7

Thanks for the breakdown of events per path, that table is the clearest statement of this I have found anywhere.

We are on the second row of it: an existing Jira Connect app adopting a Forge manifest while keeping every module under connectModules, so Connect to Connect-on-Forge. Our app is a security information provider, and all of our per-tenant data is keyed on clientKey, so getting installationId recorded for existing installs is the thing the whole migration rests on. We already declare jira:lifecycle with installed and uninstalled under connectModules, which from the other thread on this seems to be the piece people miss.

Two questions, if you can.

Does that first row hold for automatic updates as well as admin-initiated ones? Our Marketplace listing has autoUpdateAllowed: true, so we expect most of our install base to take the Forge version with no admin action at all. The documented note that installed no longer fires on automatic updates is what worries us. If the Connect /installed hook fires only when an admin clicks Update, most of our tenants would never get a mapping row.

If it does not fire on automatic updates, what is the recommended way to obtain installationId for an existing Connect tenant? The reserved connect_client_key_[uuid] property looks like it runs the other way, telling a Forge app its Connect clientKey from inside a Forge invocation. On the Connect-on-Forge path there are no Forge invocations, so as far as I can tell it is not available to us. Happy to be told I have that backwards.

Also, did the FRGE-2276 fix reach production? It does not affect our path directly, but it will when we go from Connect-on-Forge to native later, and we would rather plan around the fixed behaviour.

8

Hey @AishwaryaSabane, thanks for reaching out!

Yes, both cases result in the Connect /installed hook firing. Note, we recently updated Retrieving the Connect clientKey in Forge to emphasise that

Keeping a Connect module in your manifest means that part of your app is still built on the Connect framework, so it remains liable for the higher Connect Marketplace revenue share rate.

The avi:forge:installed:app event will fire when a Connect installation upgrades to either Connect-on-Forge or pure Forge. You can leverage this as described: Retrieving the Connect clientKey in Forge.

Something not explicitly called out in that doc that perhaps should be:

During app installation, the Atlassian account used by your app to authenticate with Atlassian APIs is granted various permissions and group memberships for the site it is being installed into. This process is eventually consistent, and the installed event may be sent before the app is granted the permissions needed to call certain APIs using .asApp(). For large sites, it may take a very long time (minutes) for your app user to be fully granted the requisite permissions.

As such, we recommend coding defensively when using .asApp() in an installed trigger. If you receive an unexpected authorisation error when calling an Atlassian API (typically a 401 or 403 HTTP status code), you should use Forge’s retry handling to reattempt the request after a delay.

from https://developer.atlassian.com/platform/forge/events-reference/life-cycle/#making-api-requests-during-installation

Both Jira’s get app property API and Confluence’s get app property API are accessible with Connect’s JWT auth token and Forge’s Oauth token. Note they have slightly different scope requirements between Jira + Confluence (unsure what the history is here…), so watch out for that.

Therefore, if you’re planning a Connect-on-Forge stage, you can access the app property APIs with either Forge’s Oauth token or Connect’s JWT auth if your Forge manifest specifies:

app:
  ...
  connect:
    key: <your_Connect_addon_key>
    authentication: jwt
    ...

Yep, has been fully rolled out to production.

Let me know if any of this needs further clarification.

9

Thanks Mitch, that is very helpful. Two things to confirm and one data point back.

First, a contradiction I would like to resolve. You say both automatic and admin-initiated updates fire the Connect /installed hook. Atlassian support told us the opposite this week, in writing:

Do not rely on the installed lifecycle event to build your clientKey to installationId mapping for your existing install base. […] Automatic updates (your listing has autoUpdateAllowed: true) do not reliably invoke the Connect lifecycle callbacks, so the majority of your install base would not produce a mapping this way.

These cannot both be right, and the answer decides whether we build a backfill for our whole install base or none of it. Our own observation covers only an explicit install, which is the case you both agree fires. Could you and the support engineer reconcile this? I am happy to pass the ticket reference along if that is easier.

Second, how is the uuid in connect_client_key_[uuid] obtained? You note the app property APIs accept Connect JWT as well as Forge OAuth, which would be excellent for us because our app is Connect-on-Forge with no Forge functions at all, just connectModules and a remote pointing at our own service. If we can read the property from that remote using the Connect JWT we already hold, we need no Forge compute.

What I cannot see is how we learn the [uuid] part of the property name without a Forge invocation. If that uuid is the Forge installationId, then an app that already knows its installationId does not need the property to discover the clientKey, and an app that does not know it cannot construct the property name. That reads circular from where I am sitting, so I assume I am missing something.

Concretely: can a Connect-on-Forge app with no Forge modules read this property purely from its remote using Connect JWT, and if so how does it derive the property name?

A data point on FRGE-2276. We migrated on 1 October, after the rollout, taking the Connect to Connect-on-Forge path rather than pure Forge. Same app.connect.key, connectModules retained including jira:lifecycle. No uninstalled webhook fired. We saw a single Connect /installed carrying clientKey, sharedSecret and installationId, and nothing else. Our uninstall handler is destructive, it deletes the customer’s linked workspaces, and the workspace survived the upgrade intact. Caveat: that was an explicit forge install of a development environment onto a site we own, not a Marketplace-distributed upgrade, so it may not exercise the same path.

Minor: on revenue share, our app is free and public, so I assume the higher Connect rate is irrelevant for us. Worth confirming in case it affects anything beyond revenue.

Thanks also for the note on eventual consistency of app account permissions during installation. That is not in the doc and would have cost us time if we ever add a Forge install trigger.

Thank you!

10

Please provide the ECOHELP ticket, I’ll take a look into it and get back to you.

The reserved key is literally connect_client_key_019cdff3-8bfb-71fe-9628-875b700aebb8 - the UUID has no special meaning, we just added the UUID to remove the possibility of a clash with any app that may have already used the property key connect_client_key.

Something along the lines of

fetch('https://<siteUrl>/rest/atlassian-connect/1/addons/<connect_addon_key>/properties/connect_client_key_019cdff3-8bfb-71fe-9628-875b700aebb8', {
  method: 'GET',
  headers: {
    'Authorization': `JWT <Connect_JWT_token>`,
    'Accept': 'application/json'
  }
})

Good to hear it’s working as intended, thank you!