Optimizing Triggers and High-Volume Data Flows in Atlassian Forge

Forge Forge Rookies
, , , ,
1

This technical note synthesizes the latest Forge documentation to present a design and implementation strategy for Forge apps that must process large data volumes (tens/hundreds of thousands of issues, comments, or change events). We focus on: (i) manifest.yml optimizations that reduce module bloat and invocation pressure, (ii) event-first architectures that cut load before code runs, and (iii) throughput-safe JavaScript patterns (bounded concurrency, back-pressure, pagination, idempotency) that respect Forge’s platform quotas. (Atlassian Developer)

1) Platform constraints that drive design

Forge enforces ceilings you cannot “optimize away,” so your manifest and runtime plan must conform:

  • Invocations: 1,200 function invocations per sliding minute per app. Design to avoid unnecessary calls (pre-filters, batching). (Atlassian Developer)

  • Runtime: typical functions 25 s; web-triggers 55 s; scheduled triggers & async-event consumers may run up to 900 s by setting timeoutSeconds (default 55 s for long-running modules). Single outbound request timeout for long-running functions is 180 s. (Atlassian Developer)

  • Network egress: 100 requests per runtime minute per invocation (rounded up with timeoutSeconds), excluding requestJira/requestConfluence. Concurrency control is mandatory under high load. (Atlassian Developer)

  • Async Events (per installation): ≤ 50 events per request, ≤ 200 KB total per push, ≤ 500 events/min, optional delay up to 15 min; retry metadata limits apply. (Atlassian Developer)

  • Trigger filters: ignoreSelf (Jira-only) and expression (subset of Jira Expressions) gate events before your handler runs; onError controls behavior on expression failure. (Atlassian Developer)

  • Product event payloads: events > 200 kB are not delivered. Plan for lean payloads. (Atlassian Developer)

These limits reward architectures that discard early, batch sensibly, and bound concurrency.

2) Manifest-level optimizations for high volume

2.1 Consolidate modules; filter at the edge

Declare one trigger with a wide events set and a precise filter; optionally one scheduledTrigger for batch windows. This lowers cold starts and centralizes back-pressure. The filter supports ignoreSelf and an expression built from a Jira-Expressions subset. (Atlassian Developer)

# manifest.yml — one trigger + one scheduled trigger
modules:
  trigger:
    - key: unified-jira-trigger
      function: unifiedHandler
      events:
        - avi:forge:installed:app
        - avi:jira:created:issue
        - avi:jira:updated:issue
        - avi:jira:deleted:issue
        - avi:jira:commented:issue
        - avi:jira:assigned:issue
      filter:
        ignoreSelf: true
        expression: >
          ["OPS","ENG"].includes(event.issue?.fields?.project?.key) &&
          event.issue?.fields?.issuetype?.name !== "Sub-task"
        onError: RECEIVE_AND_LOG

  scheduledTrigger:
    - key: hourly-batch
      function: unifiedHandler
      interval: hour

function:
  - key: unifiedHandler
    handler: src/index.unifiedHandler
    timeoutSeconds: 900   # only where long windows are needed

  • ignoreSelf prevents loops from your own writes; expression blocks noise (by project/type/etc.) pre-invocation. (Atlassian Developer)

  • Scheduled triggers start ~5 min after deployment, then follow the chosen cadence; keep them ≤ 5/app (only one may use fiveMinute). Implement idempotent jobs because duplicates can occur. (Atlassian Developer)

  • Keep under the 100 modules/app ceiling—another reason to group events under one trigger. (Atlassian Developer)

2.2 Egress allow-listing & environment variables

For high-volume integrations, explicitly allow domains in permissions.external.fetch.backend (or use Remotes for a companion service). Avoid broad allow-lists; parameterize environment-specific bases via manifest variables resolved by the CLI at deploy. (Atlassian Developer, go.atlassian.com)

permissions:
  external:
    fetch:
      backend:
        - https://api.example.com
        - ${ANALYTICS_BASE}

environment:
  variables:
    - ANALYTICS_BASE

Note: manifest variables are expanded by the Forge CLI at build/deploy time; runtime Forge environment variables are managed via forge variables .... (Atlassian Developer)

3) Event-first architecture for large tenants

3.1 Early rejection path (no redeploys)

Large tenants need dynamic rules (by project, label, SLA). Because manifest filter.expression cannot read storage, perform a runtime gate in the handler:

  • JQL boolean gate with POST /rest/api/3/jql/match to test whether the event’s issue matches a stored rule—cheaper than a full search. (Atlassian Developer)

  • If you need richer predicates, use Jira Expressions APIs (enhanced evaluate) for pre-checks in code, not in the manifest. (Atlassian Developer)

3.2 Batching and back-pressure with Async Events

When a single event would fan out to thousands of reads/writes, enqueue compact work items (IDs + intent) and process them via Async Events with timeoutSeconds: 900. Respect 50 per push, 200 KB payload, 500/min, and optional delay ≤ 15 min to smooth peaks. (Atlassian Developer)

3.3 Retries without user-managed queues

For transient upstream failures (HTTP 429/5xx), ask the platform to retry by returning an InvocationError({ retryAfter, retryReason, retryData }) from a trigger or consumer. Product events can be retried up to four times, retryAfter ≤ 900 s. Read event.retryContext to handle backoff or give up. (Atlassian Developer)

4) High-volume read patterns (Jira REST)

  • Prefer POST /rest/api/3/search (GET is being removed); use fields to narrow payloads; avoid expand=changelog unless essential. (Atlassian Developer)

  • Use /search/approximate-count to estimate result size and choose batch sizes before pulling data. (Atlassian Developer)

  • Page with startAt/maxResults, partition by project/time/key ranges for resumable scans. (General search group guidance.) (Atlassian Developer)

5) Throughput-safe JavaScript (Node 20/22 on Forge)

5.1 Concurrency limits (avoid N×100 req/min explosions)

A minimal semaphore caps in-flight calls to respect the 100 egress requests/min per invocation ceiling (note: requestJira/requestConfluence are excluded from this count, but external fetch is not). (Atlassian Developer)

// utils/p-limit.js — tiny, dependency-free concurrency limiter
export function pLimit(max) {
  let active = 0, queue = [];
  const next = () => { active--; if (queue.length) queue.shift()(); };
  return (fn) => new Promise((res, rej) => {
    const run = () => {
      active++;
      fn().then(v => { next(); res(v); }, e => { next(); rej(e); });
    };
    active < max ? run() : queue.push(run);
  });
}

5.2 Paginated, bounded readers with aborts

// services/search.js — paginated reader with back-pressure and aborts
import api, { route } from '@forge/api';
import { pLimit } from '../utils/p-limit.js';

export async function* searchIssuesPaginated(jql, { pageSize = 100, concurrency = 3, signal } = {}) {
  const limit = pLimit(concurrency);
  let startAt = 0, total = Infinity;

  while (startAt < total) {
    const tasks = [];
    for (let i = 0; i < concurrency && startAt + i * pageSize < total; i++) {
      const s = startAt + i * pageSize;
      tasks.push(limit(async () => {
        const r = await api.asApp().requestJira(route`/rest/api/3/search`, {
          method: 'POST',
          headers: { 'Content-Type': 'application/json', 'Accept': 'application/json' },
          body: JSON.stringify({ jql, startAt: s, maxResults: pageSize, fields: ['summary','status','assignee'] }),
          // requestJira is excluded from the 100-requests/min egress quota
        });
        if (!r.ok) throw new Error(`search failed: ${r.status}`);
        return r.json();
      }));
    }

    const pages = await Promise.all(tasks);
    if (!Number.isFinite(total) && pages[0]?.total != null) total = pages[0].total;
    for (const page of pages) for (const issue of (page.issues || [])) yield issue;
    startAt += concurrency * pageSize;

    if (signal?.aborted) return;
  }
}

5.3 Timeouts, cancellation, retries

Wrap expensive I/O with AbortSignal.timeout(ms); on 429/5xx, either local retry (jittered backoff, honor Retry-After) or platform retry via InvocationError for triggers/consumers (max 4 for product events). (Atlassian Developer)

6) Idempotency & de-duplication at scale

  • Checkpointing: store a high-water mark (e.g., last processed updated) in Forge storage; resume from checkpoints to avoid re-scans. (General pattern aligned with storage quotas guidance.) (Atlassian Developer)

  • Per-entity idempotency: mark processed entities (e.g., issue properties or app storage keys) to make writes safe to repeat.

  • Event replays: read event.retryContext in async/product events to distinguish platform/app-level retries. (Atlassian Developer)

7) When to use Remotes (and when not)

If a workload truly exceeds Forge’s runtime/egress ceilings, Forge Remote lets your trigger forward to your infrastructure. Balance compliance/data-residency vs performance; some features (like Runs-on-Atlassian constraints) may differ. Use Remotes in tandem with strict permissions.external.fetch.backend. (Atlassian Developer)

8) End-to-end blueprint (large tenant)

  1. Manifest: one consolidated trigger + one scheduled trigger; precise filter; minimal egress allow-list; env variables for per-env constants. (Atlassian Developer)

  2. Event path: pre-filter by expression, then JQL gate via /jql/match using a rule from storage; early-exit on mismatch. (Atlassian Developer)

  3. Batch path (hourly): partition by time/project; paginated POST /search with fields minimization; bounded concurrency; idempotent writes. (Atlassian Developer)

  4. Back-pressure: if fan-out > budget, enqueue Async Events; consumers run with timeoutSeconds: 900. (Atlassian Developer)

  5. Resilience: on transient failures, return InvocationError from triggers/consumers (respect limits); otherwise use exponential backoff + Retry-After. (Atlassian Developer)

9) Threats to validity (what breaks scale)

  • Payload inflation: large fields or expand=changelog increase cost and may hit event/product payload caps; keep responses lean. (Atlassian Developer)

  • Schedule skew & duplicates: scheduled triggers begin ~5 min post-deploy and can duplicate—ensure idempotency. (Atlassian Developer)

  • Endpoint churn: legacy GET /search is being removed; prefer POST forms and approximate-count. (Atlassian Developer)

10) Reference implementation (dispatcher pattern)

// src/index.unifiedHandler.js — one handler for all events + schedule
import {
  handleIssueCreated,
  handleIssueUpdated,
  handleIssueDeleted,
  handleCommentChanged,
  handleAppInstalled,
  runScheduledTasks
} from './handlers/index.js';

export const unifiedHandler = async (event, context) => {
  const type = event?.eventType; // undefined for scheduledTrigger events

  try {
    switch (type) {
      case undefined:
        return runScheduledTasks(context);            // scheduledTrigger path
      case 'avi:forge:installed:app':
        return handleAppInstalled(event);
      case 'avi:jira:created:issue':
        return handleIssueCreated(event);
      case 'avi:jira:updated:issue':
        return handleIssueUpdated(event);            // includes assignee changes
      case 'avi:jira:deleted:issue':
        return handleIssueDeleted(event);
      case 'avi:jira:commented:issue':
        return handleCommentChanged(event);
      default:
        console.warn('Unhandled event type:', type);
        return;
    }
  } catch (err) {
    console.error('unifiedHandler error:', err);
    throw err;
  }
};


// src/handlers/handleIssueUpdated.js — assignee change detection via changelog
import { Queue, InvocationError, InvocationErrorCode } from '@forge/events';
import api, { route } from '@forge/api';

const queue = new Queue({ key: 'notify-reporter' });

// Detect assignee change using event.changelog (Jira 'updated' payload)
export async function handleIssueUpdated(event) {
  const items = event?.changelog?.items || [];
  const assigneeChanged = items.some(i => i.fieldId === 'assignee');

  if (!assigneeChanged) return;

  // Early runtime gating example (optional): only act if JQL rule matches
  const jql = await getTenantRuleFromStorage(); // your storage lookup
  const match = await api.asApp().requestJira(route`/rest/api/3/jql/match`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', 'Accept': 'application/json' },
    body: JSON.stringify({ issues: [event.issue.key], jql })
  });
  if (!match.ok) throw new Error(`jql match failed: ${match.status}`);
  const { matchedIssues } = await match.json();
  if (!matchedIssues?.includes(event.issue.key)) return; // drop early

  // Offload heavy work to async consumer if needed (rate-shape bursts)
  await queue.push([{ payload: { key: event.issue.key, reporter: event.issue.fields.reporter } }]);
}


// src/handlers/runScheduledTasks.js — hourly batches
import { searchIssuesPaginated } from '../services/search.js';

export async function runScheduledTasks() {
  const yesterday = 'updated >= startOfDay(-1)';
  for await (const issue of searchIssuesPaginated(yesterday, { concurrency: 3, pageSize: 100 })) {
    // do idempotent maintenance work…
  }
}

Why this works: one trigger minimizes module count and cold starts; filter.expression cuts noise before your code is invoked; schedulers/consumers get 900 s windows for heavy work; pagination + concurrency capping respects egress/latency budgets. (Atlassian Developer)

Appendix — Additional considerations for high-volume, async-heavy workloads

Throughput budgets & back-pressure
Make the budget explicit: concurrency_max ≈ floor(network_budget_per_invocation / req_per_task). Prefer batched APIs; when absent, process in small, fixed windows (2–5 in-flight). Shape traffic; if a single event would fan-out massively, throttle with a semaphore (see pLimit). (Atlassian Developer)

Cancellation and timeouts, by default
Bound time with AbortSignal.timeout(ms); propagate cancellation with AbortSignal.any([...]); fail fast near the end of a scheduled window.

Retries that respect semantics
Retry only transport/5xx/429; honor Retry-After. Prefer platform retries where available by returning InvocationError (product events retry up to 4×, retryAfter ≤ 900 s). Keep handlers idempotent. (Atlassian Developer)

Idempotency, ordering, and de-dup
Assume at-least-once delivery. Use idempotency keys (e.g., issueKey:changeId) and version/updated guards. Collapse duplicate in-flight work per key.

Memory discipline & streaming
Avoid materializing 10k+ items; stream via async generators and process incrementally. Request only required fields; avoid expand=changelog unless necessary. (Atlassian Developer)

Partial-failure isolation & fairness
Partition concurrency by tenant/project; cap per-partition lanes; use circuit breakers on sustained 5xx/timeouts.

Observability for scale
Emit structured logs (tenant, project, eventType, attempt, duration). Track p50/p95 latency, request budgets used, retry counts, and early-filter drop rates. (Forge log quotas scale with timeoutSeconds.) (Atlassian Developer)

Packaging for warm starts
Slim bundles, tree-shake, lazy-load handlers (dynamic import) so cold starts only parse the dispatcher; keep top-level code pure.

Testing async at scale
Use fake timers for deterministic backoff; property-based tests for idempotency and re-ordering; inject faults (429/5xx, timeouts) and assert early-exit/checkpoint recovery.

Data-skew & hotspots
Detect hot projects/keys and give them lower per-partition concurrency or separate lanes; add ±5% jitter to batch boundaries to avoid lockstep collisions.

References (current docs)

  • Platform quotas & limits (invocations, runtime, network, storage, async events, scheduled-trigger caps). (Atlassian Developer)

  • Trigger module (filters: ignoreSelf, expression, onError). (Atlassian Developer)

  • Scheduled trigger (starts ~5 min post-deploy; cadence options). (Atlassian Developer)

  • Async Events API (50 events/request, 200 KB/push, 500/min, delays; InvocationError & retry context). (Atlassian Developer)

  • Jira Issue Search (v3) (POST /search, /search/approximate-count, /jql/match, deprecation of GET). (Atlassian Developer)

  • Manifest variables & runtime egress permissions (CLI substitution, allow-listing, Remotes). (Atlassian Developer)

TL;DR

At large scale, capacity is a manifest problem first: consolidate triggers, filter at the edge, and explicitly budget egress. In code, gate early, page predictably, bound concurrency, and let the platform retry. Use Async Events to smooth bursts, and adopt Remotes only when Forge ceilings cannot safely contain the workload. This blueprint reflects Forge’s current runtime model and Jira REST capabilities. (Atlassian Developer)

3 Likes