Does Forge invoke an exposed REST API endpoint more than once per request?

Forge
, , ,
1

Forge app (Jira) with REST endpoints exposed via the “Expose a REST API” feature, backed by a core:function handler. We test them with Jest E2E tests — the HTTP client does a single fetch per request, no retry logic, and tests run sequentially.

Locally it’s fine, but in CI a single DELETE sometimes reaches my function multiple times with different invocationIds — sometimes the same request (same x-b3-traceid) is invoked twice ~1s apart, sometimes under a different trace id entirely. The first invocation succeeds (200); the duplicate fails because the resource is already gone.

Is this expected — can Forge invoke an exposed endpoint more than once per client request (at-least-once / retries)? If so, what triggers it, and is there a stable id (e.g. x-request-id) I can use to make handlers idempotent?

Thanks!

2

Hi @emrekaraduman,

Because you indicated this is intermediate, I suspect this is due to Forge’s at-least once delivery semantics for Forge function invocations.

Here’s some additional information I dug up using AI - it looks OK to me:

Why duplicate invocations happen

Forge runs on a distributed, serverless infrastructure. The platform may retry an invocation in several circumstances:

  • Infrastructure-level retries: If a function container fails to respond within a timeout window (e.g., due to a cold start or transient infrastructure issue), the Forge platform may re-invoke the function. This can happen even if the original invocation was actually in flight and ultimately succeeds.
  • Gateway-level retries: The Forge API gateway can retry a request if it doesn’t receive a timely acknowledgement from the function host, which is why you sometimes see the same x-b3-traceid but different invocationIds.
  • Different trace IDs: A completely new trace ID suggests the retry originated at a higher layer (e.g., load balancer or edge) before even reaching the function runtime.

This behaviour is consistent with what other distributed platforms (AWS Lambda, Google Cloud Functions) document as at-least-once semantics.

Is there a stable idempotency key?

Unfortunately, there is currently no single guaranteed stable idempotency key across all retry scenarios:

  • x-b3-traceid is consistent for retries originating within the same Forge gateway trace, but not across higher-level retries.
  • invocationId is unique per invocation instance, so it changes on every retry.
  • x-request-id may or may not be propagated consistently depending on the client.

Recommended approach: Implement idempotency in your handler

The best practice is to make your DELETE (and any other mutating) handlers idempotent at the application level:

  1. Treat “resource already gone” as success: If your DELETE handler receives a request to delete a resource that doesn’t exist, return 200 OK (or 204 No Content) rather than a 404 or 409. This way, duplicate invocations are harmless.

    const handler = async (req) => {
  const resource = await getResource(req.params.id);
  if (!resource) {
    // Already deleted - idempotent success
    return { statusCode: 200, body: JSON.stringify({ message: 'Already deleted' }) };
  }
  await deleteResource(req.params.id);
  return { statusCode: 200, body: JSON.stringify({ message: 'Deleted' }) };
};

  2. Use Forge Storage as an idempotency store: If your operation has side effects beyond simple deletion (e.g., sending a notification), you can use @forge/api storage to record completed operation IDs and skip re-execution:

    import { storage } from '@forge/api';

const handler = async (req) => {
  const idempotencyKey = `delete-${req.params.id}`;
  const alreadyDone = await storage.get(idempotencyKey);
  if (alreadyDone) {
    return { statusCode: 200, body: JSON.stringify({ message: 'Already processed' }) };
  }
  await deleteResource(req.params.id);
  await storage.set(idempotencyKey, true);
  return { statusCode: 200, body: JSON.stringify({ message: 'Deleted' }) };
};

  3. In CI tests, assert on observable state, not on HTTP status alone: Since the duplicate invocation failure is a platform behaviour, test that the resource is gone (e.g., a subsequent GET returns 404) rather than relying solely on the DELETE returning a specific status code on the first call.

CI-specific note

The fact you only see this in CI (not locally) is consistent with cold-start retries - local forge tunnel runs a single, warm function instance. In CI (and production), function containers spin up fresh and are more likely to hit the retry window.

3

Hi @dmorrow ,
This seems like a hallucination.

4

Deletes are easy to handle, but I can’t be the only one who thinks that double processing for REST APIs is not acceptable.

Using Forge KVS – which has a limit of 4k requests per minute – to deduplicate requests is yet another burden on the developer and yet another case in which we have to use the limited quotas we get to work around platform issues.

I know that exactly-once delivery is hard, but just like we all used to handle that on Connect, if Atlassian wants to offer a platform, Atlassian should handle that without shifting the onus on the marketplace vendors.

Developers do not care and should not even want to know how Forge is implemented, and the principle of least astonishment should apply: if a 5$/month Heroku dyno doesn’t get duplicate requests, nor does a t1.small EC2 container, why would we ever expect that we have to deduplicate incoming requests to a REST endpoint?

At the very least, this should be highly prominent in the documentation.

5

@dmorrow, I don’t think FRGE-391 is about this topic. Wrong link?

6

This double processing seems rather scary and a big burden on developers. The proposed solution with first a get then a delete also seems optimistic at best. I’ve tried to use the Forge KVS in the past and if two read/write combinations happen in a narrow window, I believe it was around 300ms, the second read was not yet aware of the first write. There also does not seem to be a way to lock records from editing while another is writing to it.

7

I think it would be good to identify the two different distinct issues in this thread:

  1. AWS Lambda idempotency requirement
  2. Dealing with Idempotency in your code

The first one is well documented: AWS simply does not offer only-once execution for Lambda functions. Not even for HTTP requests. The Lambda architecture design documentation is clear about the requirement for Lambda functions to be idempotent. This is an AWS implementation level limitation that we simply cannot fix.

The second issue is: given that there is an idempotency requirement, how do you deal with this in your code.

I think some of the confusion here comes from the fact that Forge has forced a lot of developers to a Serverless architecture which they may not have had to deal with in their own Connect solutions. I also think Atlassian has not given enough attention to this.

In addition, Atlassian has also abstracted away a lot of the logic that allows developers to deal with persistent storage in an environment that requires idempotency. We can’t control the underlying DynamoDB configuration for Forge KVS. So we need to hope that Atlassian implemented idempotency correctly to ensure that attributes are updated correctly.

It would be great if someone from the Forge team can shed their light on how Atlassian persistent storage deals with idempotency. Perhaps this is something you can ask the team @dmorrow?

8

If there are multiple invocations of endpoints, this will be expensive to deal with for vendors. We have to pay per function call with Forge/lambda, and Atlassian shoves extra costs on us.

A second, and possibly more urgent concern is about rate limits. If our functions get called multiple times per invocation, our apps hit rate limits much faster. And there is nothing you can do about this as a vendor.

I think Atlassian should deliver a better architecture for us vendors and developers.

9

@marc, @AndreasEbert, yes, I missed the incorrect link to FRGE-391 - it was an AI confabulation so I’ve removed it to avoid further confusion.

@marc , I see your point about the additional impact on costs and rate limits, but I assume the percentage of superfluous invocations is negligible.

@remie makes a good point about separating the issue into two aspects and essentially focussing on tactics to deal with this in app code because I doubt it would be possible for Forge to eliminate superfluous deliveries.

10

For the record: I never said this has to be solved in app code. It can also be solved by Atlassian, for instance by following AWS recommendation to implement response caching:

You can implement idempotency in Lambda functions by using a DynamoDB table to track recently processed identifiers to determine if the transaction has already been handled previously. The DynamoDB table usually implements a Time To Live (TTL) value to expire items to limit the storage space used.

11

Quick update — we found a deterministic cause in our case. The duplicate invocation is triggered by calling internalMetrics (counter incr/decr) inside the handler’s request path: every endpoint that calls it is invoked twice, every endpoint that doesn’t is invoked once. Removing the metrics call makes the duplication disappear; adding it back brings it straight back.

The counter calls are synchronous, so this isn’t an un-awaited promise in our code — it looks like the metrics flush at invocation teardown delays the completion acknowledgement enough for the gateway to re-invoke, which matches @dmorrow’s “gateway retries when it doesn’t receive a timely acknowledgement”.

A couple of notes:

  • It reproduces with a single manual DELETE (not just in CI) on the deployed app — the first invocation deletes and returns 200, then ~1s later a second invocation returns the “does not exist” error, and that second result is what the caller receives even though the delete succeeded. The first invocation returned its 200 quickly, so it really isn’t about handler duration.
  • It’s not specific to “Expose a REST API” — the same happens through a UI invoke() (resolver) call: the browser issues a single request, but the function runs twice and the retried (error) result is what the user sees.
  • forge tunnel masks it entirely; only a deployed environment reproduces it, which points to a cloud-runtime teardown behavior rather than anything in app logic.

We’ve mitigated it by making deletes idempotent (treating “already gone” as success), as suggested — so the re-invocation returns success and the caller no longer sees a spurious error. That hides the symptom but not the extra invocation, so the cost / rate-limit concern raised above still stands.

Question for the Forge team: is it expected that using internalMetrics inside an apiRoute/resolver handler causes the platform to re-invoke idempotent requests (the teardown metrics flush leaving the invocation “not cleanly acknowledged” within the gateway window)? And if so, is the recommendation simply to keep internalMetrics out of the synchronous request path?