Can you update/expand the guidance on maintaining cross-product Forge Marketplace apps?

Hi Team,

The current FAQ on What if my app supports multiple products? states that the Forge platform “supports and encourages developers to build apps that are compatible with multiple products”, though not yet for distribution via the Marketplace:

[…] If your app is compatible with multiple apps, you’ll need to create two Forge apps using the same code base, and publish two separate listings on the Marketplace.

We have followed resp. EAP advice to use the identical code base for Jira and Confluence and generate two manifests from a template, with the only differences between them being app name and ID, because the Forge runtime supposedly removes inapplicable scopes automatically.

This worked fine for a while, but yesterday we had our first (likely multi-dimensional) incident that effectively caused an app outage.

While not yet fully understood, the incident seems correlated to the (highly welcome) egress permissions rollout on Monday, which triggered a new ‘Allow access’ request for users. This grant flow worked in Jira, but always resulted in an error in Confluence (other users also encountered this).

We managed to work around the problem by explicitly removing the Jira scopes from the Confluence manifest, and instructing the linter to ignore the Jira specifics in the code based via --no-verify during deployment.

Both of these kludges would require a refactoring towards a likely more complex CI/CD pipeline, and might even require splitting inapplicable code paths from a product’s code base one way or another.

So before we go down this route, we would highly appreciate updated guidance on how to organize a code base for cross-product Forge apps distributed via the Marketplace - specific questions/asks seem to be:

  1. Is the Forge runtime still supposed to transparently handle cross-product deployments (i.e. removing/ignoring inapplicable scopes/code), which would suggest that we might have simply hit a regression or edge case?
  2. If that is not the case anymore, can you suggest alternative approaches, both short- and mid-term?
  3. Can you maybe provide a cross-product Marketplace sample app with CI/CD to make this easier for others developers down the road?

Many thanks,
Steffen

9 Likes

Hey @sopel,

I appreciate you taking the time to discuss the details of this with me offline. This is a rather complex problem.

The core of the problem here, to the best of my understanding, is the following:

  • The egress controls rollout required all apps to be consented to. In the past, your app had not required consent.
  • Because scopes are defined the same across asUser and asApp calls, your asApp scopes were showing up in the consent flow (even though you weren’t doing asUser calls).
  • This lead to cross-product scopes appearing in both apps consent flows (which is what would have happened regardless of the egress rollout if you were using asUser.

Forge on Marketplace does not support cross-product deployments natively, the EAP advice you mention is a workaround.

For short and long-term solutions, I may have to defer to @rfernandes.

2 Likes

Thanks @danielwinterw,

This helped us to get a better understanding of some aspects of this indeed multi-faceted and thus rather complex issue.

We have just filed a separate issue for one aspect that now turned out to be unrelated and only amplifying the impact during the incident:

We are unable to reproduce the main issue (consent grant flow does not finish in Confluence) with a privately distributed app, so it might indeed be related to the specifics of a cross-product Marketplace app, and we obviously don’t want to mess with our production deployment to debug this.

As a result, we are kinda blocked on reinstating your team’s cross-product app deployment advice as a workaround, so getting more insights and guidance by @rfernandes would be very much appreciated :slight_smile:

2 Likes

Sorry for the delay.

Re. the overall stance on multi-product apps

Forge today has been designed in a way that isn’t product specific, so an app can be composed from functionality from multiple products. That said, this is a bit theoretical, because in practice as you’ve discovered we don’t have a great story for how a single app can be installed in and interact with multiple products.

This is an area we are interested in investigating further, but definitely needs more work and thought. The doc you reference stating that Forge “supports and encourages” this is probably too strongly worded – we’ll tone that down.

For now and the near/medium term, Marketplace apps can only be made available for installation into a single product, so it’s good to design your apps with this in mind.

Re. your immediate issue

Even though separate apps, installs and listings are required for Jira & Confluence apps (and this will continue to be true for the medium term), we do expect it to be possible to share code between the two apps. Based on your experience I think there’s 2 specific things I need to dig into – I’m essentially paraphrasing your questions, let me know if this is right:

  1. You used to be able to deploy an app to Jira (/Confluence) despite it including code and scopes for Confluence (/Jira). This stopped working at some stage, and now you need to remove the unrecognised scopes and pass in --no-verify to avoid validation errors. Was this change a change in Forge, and if so was it deliberate? (If so, our claims that code can be shared need to be more nuanced.)
  2. To what extent is the behaviour in 1. affected by whether the app is listed on Marketplace, i.e. could it be that the platform is enforcing stricter validation on listed apps? (This isn’t something you can experiment with on your side because you don’t/can’t have “dummy” listed apps to play with.)

Assuming I’ve understood that right, I’ll try to get back to you soon with answers to those 2 questions.

2 Likes

Thanks for looking into this @rfernandes!

We are delighted to hear that nothing has changed regarding your long-term cross-product app aspirations, and that you intent to retain the possibility to share code between the apps, the unexpected loss of that mechanic caught us off guard and has been a pretty painful experience. As for your questions:

  1. Your colleague @XavierCaron has meanwhile confirmed that the egress permissions rollout indeed caused a regression with the Forge Marketplace app specific cross-product scope filtering, as we suspected as per my initial question here. This has now been fixed so that we should be able to reinstate our cross-product code base and drop the --no-verify workaround - we haven’t tested that yet, but given the incident’s root cause we are confident that it will work as before :slight_smile:
  2. As per 1. the behavior is indeed exclusive to Marketplace apps, because only those require the cross-product scope filtering. Unfortunately such Marketplace specific constraints imply that it is close to impossible for partners to test let alone debug any app feature related to those few yet significant differences between listed and unlisted apps, so we remain concerned that future incidents in this context are bound to happen.

While we are optimistic that the Forge platform will be considerably less prone to incidents, we all know that at scale “everything will eventually fail over time”, so we hope your team will find a way to address the current inability to test Marketplace apps in an actual production environment (e.g. via segmented rollouts).

Cheers,
Steffen

@rfernandes

This cross-product apps incorrect documentation cause a major issue for our app (app id
794ed617-4a8d-4b14-8b70-83eced546fac
). Perhaps you can look the thread and ask somebody to have quick fix for it.