FAIL_IF_EXISTS and lazy TTL deletion: does an expired but unswept key still count as existing?

Forge
1

The KVS docs say expiry is asynchronous: “Expired data is not removed immediately upon expiry. Deletion may take up to 48 hours. […] During this window, read operations may still return expired results.” The mitigation they give is read-side, request EXPIRE_TIME and ignore values whose expireTime has passed.

I can’t find the equivalent statement for a write. If a key is past its TTL but hasn’t been swept yet, does this throw?

await kvs.set(claimKey, { claimedAt: new Date().toISOString() },
  { keyPolicy: "FAIL_IF_EXISTS", ttl: { value: 10, unit: "MINUTES" } });

FAIL_IF_EXISTS plus a short TTL is the obvious way to build a lease. If a ghost key still counts as existing, a 10 minute lease can hold for up to 48 hours after it expires.

I know the custom entity route from 97917. The accepted answer there puts a version attribute on an entity and compares it inside transact().set(), which closes the takeover race without depending on TTL. The plain-key equivalent is what I can’t find. The KVS transactions page documents ttl and no conditions, so compare-and-swap seems to mean moving the claim into a custom entity and giving up keyPolicy entirely.

Inspecting the conflict isn’t an option either. I unpacked the published @forge/kvs tarball (2.0.3): returnMetadataFields appears on exactly one options variant, OverrideAndReturnSetOptions, which pins keyPolicy: 'OVERRIDE', and the overload that accepts keyPolicy at all returns Promise<void>. After the throw you have to re-read the key, which is a second round trip into the window you were trying to close.

That is what our own code does, in LeanZero’s Forge rule engine: the invocation claim function. We assumed the ghost does conflict, so on the error we re-read and compare claimedAt instead of trusting the TTL. I posted the flat version of this on 101889 in July, that a TTL isn’t a dedup window because deletion is lazy. That was about reads. I don’t know whether it holds for a conditional write, and I haven’t run it: a sweep that happens to fire on a dev site isn’t a contract, and this is the kind of thing that could change without a note.

Is an expired but unswept key visible to FAIL_IF_EXISTS?

2

Hi Mihai, I’m from the Forge Storage team.

Yes, an expired but not-yet-swept key is still visible to FAIL_IF_EXISTS, so the write will fail until the key is deleted. TTL should therefore not be relied on for precise lease semantics.

As a best-effort workaround, the app can read the existing value after a conflict, check its logical expiry, and overwrite it if expired. This is not atomic, so for strict mutual exclusion, the current recommendation is the custom entity transaction approach you described.

We’re looking into additional conditional-write support for KVS and will review the documentation to make this behaviour clearer.

3

Hi @JerryZhao, thanks for the direct answer. That settles it, and it’s the one I was hoping not to get.

Checking our own code against it, the exposure is narrower than I feared. Where the payload carries an executionId or a changelog id we key the claim on that, so that key isn’t reused and a ghost does no harm. Only our windowed rule-plus-issue fallback leans on the claimedAt comparison, and that is where I’d take your recommendation and move to a custom entity.

One more thing that would help while the conditional-write work is in flight. If the KVS page gets a write-side note, could it say whether “up to 48 hours” is a hard bound or a typical case? A lease fails closed for that whole window, so I think the number decides whether this is a workaround or a stuck app.