I’m building a Forge app for Jira that needs to do a full “sync” across all issues in an instance — reading each issue, computing hierarchy parent/child relationships, and writing entity properties back. On a mid-size instance (~3,000 issues) this already strains Forge’s limits. On a large instance it’s essentially impossible within the free tier.
Here’s the architecture I landed on after several iterations, and the problems I’m still hitting:
The two-phase queue approach
Since Forge functions have a 55-second hard timeout, you can’t loop over issues in a single invocation. I use a Forge Queue with a two-phase pattern:
- Collect jobs — fetch one page of issues from Jira’s REST API (50 at a time), split them into mini-batches, push batch jobs + the next collect job to the queue
- Batch jobs — reconcile ~3 issues per invocation (read the issue, compute hierarchy, write entity properties back), update a progress counter
The queue consumer dispatches based on a type field in the job body.
The problems
1. 429 rate limiting from Jira’s REST API
Each batch job makes multiple REST calls per issue (read issue, read parent, write properties). Jira starts returning 429s almost immediately. I added retry logic with exponential backoff (5s → 15s), but:
- Each retry burns precious seconds from the 55s function timeout
- With 500ms forced delay between issues to avoid hammering the API, a batch of 3 issues already takes 1.5s minimum — and that’s without any 429 retries
- On a real run, 429s turn a 3-issue batch into a 20-30 second invocation
2. The 1000 cyclic invocation limit
Forge limits queue jobs that push more queue jobs to 1000 push() calls per app (per hour? per day? — docs aren’t entirely clear). Each collect job counts as one push. With 50 issues per page:
- 3,000 issues → 60 collect jobs → 60 push calls (fine)
- 50,000 issues → 1,000 collect jobs → hits the limit exactly
- Anything above 50,000 issues simply cannot complete within this limit
I already batch up to 50 events per push() call (which helps), but the push calls themselves are what count.
3. Free tier exhaustion from testing
Testing the sync against a real instance with a few thousand issues completely consumed my free-tier function invocation quota. Each collect + batch job pair is an invocation, and with BATCH_SIZE=3 there are a lot of invocations. I can’t iterate on the sync logic without burning through the free tier.
4. CAS conflicts on progress tracking
I track sync progress in Forge KVS (custom entities) with optimistic concurrency (CAS). Multiple batch jobs run concurrently and all try to increment processedIssues. With retries this usually works, but under heavy load I occasionally lose increments (>5 CAS retries exhausted).
What I’m considering
- Increasing PAGE_SIZE — fetch more issues per collect job to reduce the number of push calls. But this makes each collect job slower and risks the 55s timeout on the collect side.
- Increasing BATCH_SIZE — process more issues per batch invocation. But with 429 retries and API latency, 3 is already tight.
- Switching to web triggers — use an external scheduler to drive the sync via web triggers, avoiding the 1000 queue push limit entirely. But this feels like fighting the platform.
My question to the community
How are other developers handling bulk operations across thousands of Jira issues in Forge? Is there a pattern I’m missing? Are there plans to increase the 1000 cyclic invocation limit or provide a “batch processing” primitive in Forge? Has anyone successfully synced 10,000+ issues without external infrastructure?
Any war stories or suggestions would be hugely appreciated 