We run a Forge app for Jira (@forge/kvs 1.5.0). Our API metrics dashboard shows a steady stream of 4xx, all 404, on the “unknown” Request URL row under the Atlassian source. No 5xx, app is healthy, sync works fine - but the 4xx count looks alarming and there was no obvious cause.
I spent a while tracking it down and want to confirm the behavior:
Ruled out our own REST calls. I wrapped every requestJira call to log any non-2xx response. Zero hits, including straight through a 404 spike on the dashboard. So it is not our Jira REST traffic.
Confirmed it is KVS. The app only calls two Atlassian APIs: @forge/api (requestJira) and @forge/kvs. I temporarily logged every kvs.get / kvs.getSecret that resolved undefined. The correlation was exact: in one hour the dashboard reported 4xx = 101, and our missing-key read count was 101.
So kvs.get(key) on a key that does not exist returns undefined to our code (as documented), but the underlying storage GET appears to return a 404, and that 404 is counted in API metrics as an “unknown” client error. About 95% of ours are normal read-before-write existence/dedup checks, where a miss is the expected path (first time we see a comment, etc.).
Questions:
Is it expected that a missing-key kvs.get 404s at the transport layer, and that this is counted as a 4xx in API metrics? The GraphQL storage API returns null for a missing key - does the REST gateway the SDK now uses return 404 instead?
If so, can missing-key GETs be excluded from the API-metrics 4xx (or surfaced as a distinct, expected outcome)? As it stands they bury genuine 4xx - e.g. a real deleted resource - in noise and make the dashboard look unhealthy when it is not.
Is there a recommended “does this key exist” pattern that avoids the 404 and is not eventually-consistent like query()? (We need strong consistency for dedup, so query is not a safe substitute.)
Hi Jakub,
Thanks for the thorough investigation and clear write-up - we really appreciate you taking the time to narrow this down so precisely.
To answer your questions, you are correct on all counts:
Yes, this is expected behaviour: The @forge/kvs SDK uses our REST API under the hood (rather than GraphQL). A GET on a key that doesn’t exist returns a HTTP 404 which becomes undefined via the SDK - this is the technically correct HTTP response for "Resource Not Found". However, unlike a traditional 404 (e.g. a deleted or invalid resource), this is the expected outcome in a read-before-write pattern like yours.
The metrics issue is a fair point: We aren’t in a position to modify ‘what is a valid 404 error’ from a metrics perspective, because there’s no way for us to distinguish between expected and unexpected "Resource Not Found" errors. App developers will have to make those determinations themselves unfortunately and monitor their apps accordingly.
Our Recommended approach: If your pattern is read-then-conditionally-write (i.e. “only store if the key doesn’t already exist”), consider using Conditional Set logic instead. This lets you atomically set a value only if it doesn’t already exist, and avoid the separate GET request entirely. This both eliminates the 404 noise and reduces the number of API calls required as well as ensuring atomicity. For your existing read-before-write checks where you genuinely need the value (not just existence), the 404s are harmless and can be safely ignored in your metrics interpretation for now.
We’ll update the documentation to make this behaviour clearer so other developers don’t hit the same confusion.
Thanks again for raising this!
Thanks, that all makes sense and the docs update will help.
One follow-up on the metrics side. I get that you can’t distinguish an expected missing-key read from a real “resource not found” - that’s our call to interpret. But could the source be attributed correctly instead? Right now KVS calls land in the “unknown” Request URL bucket because there’s no route template for the storage endpoint, so they’re indistinguishable from genuinely unmatched product calls.
If KVS/storage traffic were labeled as its own route or source (the way the dashboard already separates Atlassian vs non-Atlassian), we could filter it out and get a clean 4xx view for our actual product API calls - and a genuinely unexpected 404 wouldn’t be buried in storage noise. That’s just labeling traffic you already know the origin of, without you having to judge whether any given 404 was “expected.”