RFC-8 Forge Remote

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building shared understanding of a topic. It expresses a technical solution, but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines : keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Summary of Project:

The Forge Remote project aims to make it easier for developers to securely integrate their Forge apps with remote backends. This post explains our approach to tackling the problem including some high-level technical details. We’re looking for feedback to make sure we build the best possible solution.

  • Publish: 28 April 2023
  • Discuss: 17 May 2023
  • Resolve: 29 May 2023

Problem

Forge allows you to build fully within the Atlassian platform, but there are plenty of valid reasons why you might need to connect your Forge app to a remote system:

  • You have hit the limits of Forge’s native capability (e.g. you process a lot of data)
  • You’ve already built a back-end stack that works for you, and you’d like to leverage it with new apps that you build on Forge
  • You need to connect with best-of-breed technology (e.g AI services)
  • You need to integrate with another SaaS product or platform

With Forge today it’s possible to interact with remote systems but it can be difficult and inefficient. For example, you can set egress permissions to make remote Fetch API calls but there’s no easy way to authenticate those requests. Likewise, you can use Web Trigger functionality to invoke your Forge app remotely but you need to craft your own auth solution.

FRGE-1167 We are in the process of implementing authentication between our Forge App and our backend, and it is a pain in the …

This means wasted development time that could be better spent building valuable features.

Who are we solving for?

Forge Remote is the solution for Forge apps that need to integrate with remote services. It’s an option for developers willing to take on the additional complexity and security responsibilities of managing a remote service. For most use cases, we’d still encourage apps built on native Forge capabilities where possible.

Connect-on-Forge will remain the pathway for existing Connect apps to migrate to Forge. There are still a number of blockers we need to address before this becomes a viable option for many apps and these will be addressed as part of a separate initiative.

Proposed Solution

There are several different ways that a Forge app might need to communicate with a remote service, and we plan to address these in a series of milestones. Broadly speaking, we’ll be expanding the concept of remotes which already exists in the Forge Manifest and applying that to support more use cases.

Milestone 1: Remote back ends for Forge front ends

In this milestone, we’ll make it easy to connect your Forge-hosted front ends (Custom UI or Client-side UI Kit) with your remote back ends. You will be able to specify remote endpoints in your app’s manifest and Forge will automatically route back-end requests.

These requests will include a signed token that enables the remote to securely verify the request came from Atlassian. You will be able to use any standard JWT library to do this (e.g. java-jwt or node-jsonwebtoken) with a public key which we will expose through a JWKS endpoint.

They will also include short-lived (60min) OAuth 2 bearer tokens so that the app can communicate back to product APIs.

A manifest for a macro with a remote backend might look something like this:

modules:
  macro:
    - key: confluence-remote-greeting
      resource: main
      resolver:
        endpoint: remote-hello-world-ui           ---> Instead of function
  endpoint:                                       ---> Instead of function
    - key: remote-hello-world-ui
      remote: remote-app
      route: 
        path: echo/
      auth:
        tokens:
          user: true                                    ---> asUser() access token
          app: true                                     ---> asApp() access token
remotes:
  - key: remote-app
    baseUrl: https://remotehost.com/`

Requests routed to that remote endpoint might look something like this:

  "access_tokens": {
    "user", // Short lived oauth token (60mins)
    "app"// Short lived oauth token (60mins)
  },
  "context_token": "...", // Signed token for verifying the requests
  "api_base_url": "api.atlassian.com/ex/<cloudid>"
  ...
}

Milestone 2: Remote delivery of Atlassian webhooks/events

In this milestone, we will use the same mechanism to route Atlassian product events directly to a remote backend. These requests will also include a signed token to verify the request came from Atlassian and short-lived tokens to call back into the product APIs.

A manifest with a trigger that sends an issue-created event to a remote backend might look like this:

modules:
  trigger:
   key: test
   events:
     - avi:jira:created:issue
   resolver:
         endpoint: remote-hello-world-ui           ---> Instead of function
  endpoint:                                       ---> Instead of function
    - key: remote-hello-world-ui
     remote: remote-app
      route: 
        path: echo/
      auth:
        tokens
          app: true                                     ---> asApp() access token
remotes:
  - key: remote-app
    baseUrl: https://remotehost.com/

And the payload would look like the previous example.

Milestone 3: Remote access to product APIs at any time

In this milestone, apps will be able to call Atlassian product APIs from their remote back ends at any time. Apps will be given a mechanism to access APIs without a user interaction or product event needing to occur. This could be via credentials sent out with an installation hook (like Connect) or some other mechanism.

Where possible we would encourage the use of the short-lived credentials (from previous milestones) but we know certain use cases will require long-lived API access. For example, a timed batch job executing on the remote or some other external, non-Atlassian event-driven application interaction.

Milestone 4: Remote access to Forge Storage (and more?)

In this milestone, you will be able to access Forge storage remotely using the auth mechanisms introduced in the previous milestones. This means you’ll be able to integrate with remote compute but store data at rest in Forge.

If there is demand, we could also include other Forge capabilities like Forge Async Events.

Milestone 5: Improved remote access to Forge functions (Web triggers)

Web triggers enable remote invocations of hosted Forge functions but the current offering is very basic. You need to solve fundamental capabilities for yourselves (like auth). As part of this milestone, we will enhance the functionality of web triggers to add flexibility and provide an out-of-the-box auth solution.

Actions

While we would appreciate any reactions you have to this RFC (even if it’s simply giving it a supportive “Agree, no serious flaws”), we’re especially interested in learning more about:

  • What do you think about the order of priorities? Are we building the most useful functionality first?
  • Are there any use cases you don’t think we’ve covered?
  • Would you have a preferred way to receive long-lived API credentials?
  • Are there any Forge capabilities you’d like remote access to other than Storage?
  • Any initial thoughts you might have on the finer details of running a remote backend e.g.
    • Releasing new versions
    • Traceability and observability
    • Throttling and retry policies

It’s our goal to have an EAP for milestone 1 available by June 2023. I’ll update this post when we’re ready to start taking sign-ups.

Thanks for your feedback!

27 Likes

This is awesome :heart_eyes: We would be able to start migrating most of our apps to forge as soon as you have reached Milestone 2 :tada:

In terms of the actions:

What do you think about the order of priorities? Are we building the most useful functionality first?
I would move Milestone 4 to either Milestone 2 or 3, or maybe even 1, as access to Forge Storage is one of the most requested features that would solve a lot of privacy/compliance headaches. It’s basically the most important thing Connect is missing.

Are there any use cases you don’t think we’ve covered?
Probably there is :sweat_smile: but for us, this seems to cover every aspect we need.

Would you have a preferred way to receive long-lived API credentials?
I don’t think a long-lived API credential is useful, as I would rather not store those in my database. I would even argue 60m is already very long as most serverless compute resources do not run this long. A possible alternative would be to shorten the timeout to 20m and also include a short-lived refresh token. If the task continues longer we can request new tokens.

Are there any Forge capabilities you’d like remote access to other than Storage?
I must admit that I can’t think of any other Forge capabilities?

Any initial thoughts you might have on the finer details of running a remote backend?
The main question I have is how this would work in terms of DX. If I run this locally, how will it work? Can I run my forge and remote resources locally and will they be able to interact? I think this is also an important aspect for being able to successfully develop co-existing Forge and Remote projects.

13 Likes

Thank you so much for sharing this very well-written and detailed RFC, @AdamMoore!

What do you think about the order of priorities? Are we building the most useful functionality first?

For our use cases, we are most interested in Milestone 1 & 3. We essentially fit your description of a non-Atlassian event-driven application requiring long-lived API access. An install hook type of flow like in Connect would be perfect for us to receive credentials for our remote backend.

The order of milestones seems reasonable to me.

Are there any use cases you don’t think we’ve covered?

I think this is related to the Milestone 3 type of use case: It would be great if from the Forge backend you could call your remote backend, using the Atlassian-controlled authentication that is part of the Milestone 3 install hook.

I may be able to handle most of my logic in the forge backend and only need to call my remote backend in certain kinds of cases. For that, it would be great if I could still use a forge function as the resolver for my UI modules. And then, in this resolver, if needed, I could make authenticated calls to my remote backend.

Another use case could be filtering Atlassian product events in a forge function, and only forwarding certain ones to your remote backend. (Sophisticated event filtering would also be nice to have in the manifest, of course. But this would allow developers to just do it themselves. :slight_smile:)

I’m thinking of adding a new type of forge fetch JS API which allows me to make authenticated calls against the remote backends I have registered in my manifest.

For what it’s worth, this should not be a blocker, as your Milestone 1 proposal sounds like it would already solve our use cases. But with this I could start building my backend in Forge, and then slowly add calls to my remote backend in my forge resolvers when I need them. All that while still benefiting from the advantages that running code in a forge function may bring. Best of both worlds. :slight_smile:

Would you have a preferred way to receive long-lived API credentials?

The install hook type of idea mentioned in Milestone 3 would be great for us. Alternatively, an API that lets us generate long-lived credentials would also be neat. From a security perspective that might even be nicer, since you could allow us to restrict the permissions of these generated credentials. E.g. in our case, these long-lived credentials would only need read permissions while the main app itself also needs write permissions.

We have use cases which may require credentials to live for at least a couple of hours (if we cannot refresh them from our remote backend).

Are there any Forge capabilities you’d like remote access to other than Storage?

I don’t think so. Even if there are any, I’d believe that with Milestone 5 you should be able to cover all of them, since I can then just call any forge function from my remote backend. :+1:

Any initial thoughts you might have on the finer details of running a remote backend?

Not at the moment.


One more thing about what you wrote here:

We are interested in this forge remote compute use case because we have Connect apps that require it. If I understand correctly, the migration pathway for Connect apps will be as follows:

Connect → Connect-on-Forge → Forge Remote

As you say though, there are a couple of blockers in Connect-on-Forge that prevent a lot of apps from migrating. You say these will be addressed in a different initiative. Will you also inform us about your plans for Connect-on-Forge in an RFC like this? Because that would be great!

As far as I can tell a lot of the ideas around Milestone 3 are similar to what Connect-on-Forge already does today. So it would be great if we could work together to minimize migration efforts between Connect-on-Forge and Forge Remote.

Cheers,
Sven

6 Likes

Thank you for this update, this sounds promising!

What do you think about the order of priorities? Are we building the most useful functionality first?
I think most use cases for milestone 2 are enabled by milestone 3, so I’d prefer to switch the order here.


While this sounds promising to vendors, I wonder how this will be communicated in the marketplace. This approach basically allows us to turn forge into connect v2. Are these APIs here to stay or are they just an intermediate approach to help vendors move forge is more feature complete? It would be very frustrating to move to these APIs just to be forced to migrate away in a year’s time.

For most use cases, we’d still encourage apps built on native Forge capabilities where possible.

Will vendors that use these APIs have significant disadvantages in the marketplace?

1 Like

@AdamMoore , This is great

For us the best options would be milestone 4 first, allowing us to store data in Forge for a forge-connect app

Regards
Paul

Thanks @Remy!

  • Fair point with Forge storage. Let’s see how the other responses go.
  • Some examples of other capabilities we’ve been thinking are async events or scheduled triggers. Scheduled triggers are pretty basic, but it might be useful to get a remote invocation periodically (e.g. every 24 hours) if there was something you wanted to run on a schedule.
  • You raise another good point about the DX and local development, it’s something we’re thinking about but haven’t settled on a solution yet.
1 Like

Hey Sven, thanks for your detailed response :smiley:

  • I’m glad you raised making remote calls from a Forge function. I especially like the event filtering use case! As a quick win, we could probably include the context token in the app context so you could craft your own authenticated fetch API requests (and verify them against the JWKS endpoint). Access tokens might be a bit trickier but we’ll have a think about it.
  • Yes, you’re right the migration pathway would be Connect → Connect-on-Forge → Forge Remote. It’s pretty clear with milestone 1 and 2, you could start using Forge Remote once you started replacing your Connect Modules with Forge ones. As you point out though, milestone 3 is where it gets interesting. We need to think more about how that transition would work. Also, plenty of other questions like how you’d map clientKeys to the relevant id in Forge. Lots more RFCs to come on these topics I’m sure :slightly_smiling_face:
3 Likes

I like this idea. This will introduce a solid way to safely integrate backends to overcome the Forge limitations. We’ve also implemented this with a few of our Forge apps. We are using a shared secret between the Forge backend (stored in Forge variable) and a GCP backend for that. Requests from the client are either secured by a per-user token that is transmitted over this backend-to-backend communication or entirely proxied.

This approach is slow. Having a baked-in solution would be great.

One thing that is missing in this RCF: Already today several Forge features are not data residency compliant (Storage) – the backend functions are even hosted in the US. By encouraging the usage of remote backends Forge Remote will shift the responsibility to handle realm-migrations, data-residency and multi-tenancy backt to the vendors responsibility. Please provide functionality to support those requirements with Forge Remotes in an early stage!

3 Likes

Good questions Philip.

The idea is definitely that these features are here to stay. We’ll continue to improve Forge’s native capabilities to support more use cases but there will always be reasons to connect with remote resources.

We are currently considering the most effective approach to inform our customers about these features. We might include some trust signals on the installation screen, similar to how we display the ACT_AS_USER scope in Connect or the egress permissions in Forge. It’s not our intention to disadvantage these apps but we do want to be transparent with customers so they can make informed decisions.

Thanks @JulianWolf

Our priority for data residency has been Connect (because that’s still where more customers are), but you’re right, this definitely increases the priority around bringing similar functionality to Forge.

I’m not sure exactly when we’ll support full data residency (inc. realm migrations etc.) but we’ll definitely keep this in mind when we’re designing manifest changes etc.

Thanks @AdamMoore even though I am a bit surprised by this statement. Not having a clear and timely roadmap for residency support in Forge will be major blocker in terms of platform adoption.

Why would I migrate a Connect application to Forge if I loose this functionality that is especially requested by large customers. Same goes for starting new apps: a Connect competitor will always be able to support those crucial features.

Forge is about „secure, reliable, and scalable“ apps - no plan for residency and realms is at least not a scalable approach.

For Forge Remotes: Can Atlassian make sure that not every request (e.g. for Macro rendering) is proxied by the US-based Forge backend to the vendors remote? This would result in a really slow experience for users, especially when the vendors remote is EU based.

4 Likes

I second the sentiment voiced by @JulianWolf.

Not having data residency is a major blocker for moving to Forge, as well as articles 14 and 15 of the Atlassian Developer Terms.

2 Likes

Thanks @AdamMoore, sounds great overall, and we agree with Milestones 1-3. As for “Let’s see how the other responses go”, I indeed disagree with Remie’s proposal to promote Milestone 4 - to the contrary, I think Sven captured things well here:

I would go even further and propose to demote Milestone 4 to be the last one, given you can indeed access Forge storage via Milestone 5 anyway?

Or put another way, having dedicated support for Forge storage (or any other Forge capability) only seems to be a DX improvement rather than delivering additional capabilities, whereas Milestone 5 unlocks all of them, or am I missing something here?

1 Like

Does this mean that we would have to configure a proxy function? Or will this be a service that the Forge platform provides? In either path - I’m concerned about the latency that Atlassian might introduce…

If it is a service in between - any chance that it would be able to do proper caching based on the content headers returned?

1 Like

I think it mostly depends on your use-case :man_shrugging:

If you are already on Forge, like yourself, and you are only looking to use Forge Remote to offload highly specific tasks it might not be very important to have access to Forge Storage from the Forge Remote. You already have access to Forge Storage from your Forge functions, and you either pass the data on or call a web trigger to retrieve the data (if even required).

However, if you are coming from Connect, the lack of secure storage within Atlassian infrastructure is crippling.

We will probably be using Forge Remote to keep 95% of our app on our own infrastructure and will only begin with static hosting on Forge. Having access to Forge Storage from our own infrastructure will allow us to move more quickly and achieve better trust signals with our customers as their data will remain on the Atlassian infrastructure whilst we can steadily migrate to Forge.

So yeah, I would say it’s just different perspectives.

Hi everyone, regarding Forge data residency, this is currently work-in-progress. I mentioned in the November update that one of the key dependencies we have to rollout Forge data residency is enabling multi-region compute. We are currently doing some heavy lifting to get multi-region compute out, so we can ship data residency of hosted storage. I would like to emphasise that Forge data residency is one of our top priorities. As we are still early in the project, keep an eye on the public Trello card for updates. Thanks!

3 Likes

Hey @AdamMoore thanks for writing this up.

Priority order:
I think ScriptRunner would need milestones 2 and 3 first so we can do ingress and egress, maybe including some of the event filtering that @SvenSchatter mentioned, then 1 and 5 so our UI can be native Forge and any business logic that fits Forge functions better than our Connect implementation can be migrated to Forge but still be called by remaining Connect services, finally followed by 4.

Other use cases:
Like @SvenSchatter said (again) “It would be great if from the Forge backend you could call your remote backend, using the Atlassian-controlled authentication that is part of the Milestone 3 install hook.”

Preference for receipt of long-lived API creds:
Install hook is good, even if we have to refresh them with some frequency

Other capabilities:
Not that I can think of

Finer details:
For milestone 2 (event delivery) I’d expect the same retry logic and rate limiting of outbound requests from Atlassian to apply to this implementation as it currently does to the Connect implementation.
For milestone 3 (product API access) again I’d be expecting the same cost budget + concurrency rate limiting logic that is currently in place.
For milestone 4 & 5 (Forge storage and function access) I’d want to know what the rate limits would be to help identify whether moving data/compute out of Connect would help or hinder the app.

Anything that can be consumed by a Connect app I’d expect to have a long lived identifier or URL for - ie dont change it when a new version of the Forge app is deployed because that would be a nightmare to keep in sync with Connect app deployments.

Anything that the Forge platform is doing to communicate externally (eg milestone 1 and 2) I’d expect there to be well thought through error messages with plenty of metadata and for those to be visible both in the Forge logs but also available somehow for consumption by external log/metric processing tools. The existing Forge observability tooling just doesn’t match the capabilities we have with other tooling in our existing Connect app hosting setup.

6 Likes

In addition to the post by @SushantBista’s, it’s worth mentioning that the service responsible for making requests to remote backends is also the one that invokes Forge functions. This means that as we introduce multi-region compute support in Forge, we’ll be able to extend it to Forge Remote as well.

Once both projects make further progress, we will have a better understanding of the exact timeline for when we can provide support.

1 Like

@sopel you’re right, but the other thing to consider is latency. If your app is heavily dependent on communicating with storage then having to invoke a Lamba each request might be a problem.

It’s similar to @SvenSchatter’s point that you don’t need milestone 2 if we added a “Forge function to Remote” option instead.

It’s clear there’s a good use case for all of these options, a little harder to say which is most important :upside_down_face:

2 Likes
  • What do you think about the order of priorities? Are we building the most useful functionality first?

For us - Milestone 4 will probably remove the most blockers (in combination with Milestone 1) - so we would prefer it as the second priority. However, for us - it would not actually be useful until either milestone 2 or 3 is completed. So perhaps the overall priorities are correct (for our apps at least)

  • Would you have a preferred way to receive long-lived API credentials?

We have use cases for our apps where we would like to call Atlassian API’s on a schedule - that could be days or even months apart. However, we do not want to store very long lived credentials if we can avoid it.

For example, a timed batch job executing on the remote or some other external, non-Atlassian event-driven application interaction.

So the point mentioned about scheduled delivery of access tokens - for us this would be ideal as we would if we could control the schedule. This would allow us to not store the access token beyond the scope of the actual scheduled task we are executing for the user.

  • Any initial thoughts you might have on the finer details of running a remote backend e.g.

For us - we would like to emphasise @SvenSchatter 's point:

As far as I can tell a lot of the ideas around Milestone 3 are similar to what Connect-on-Forge already does today. So it would be great if we could work together to minimize migration efforts between Connect-on-Forge and Forge Remote.

What I perceive is that this will result in quite a confusing landscape for developing cloud apps for the Atlassian platform. Developers will have the choice between Connect, Connect on Forge, Forge, Forge with remote backend and OAuth…

This does not feel sustainable. Perhaps some clear outline of which developer platforms are here for long term and which should be viewed as a step in migration might be very helpful.

2 Likes