Situation: In our app we use @forge/llm for data-crunching; the prompt’s output must be a specific JSON format so our code can process it. Currently we “enforce” this by having a textual rule in our prompt, and hope that the AI always follows that rule.
Ask to Atlassian: We learned that Claude models support proper enforcing of output-format via Structured outputs (their name). They specifically mention that this can enforce JSON output of a given json_schema. We’d like to use that. Can you add support for that in @forge/llm? What’s your thinking on that?
Hi @AndreasEbert, it isn’t there, and I’d keep pushing for it. Nothing in the shipped types of 1.0.3 or 1.0.4-next.0: no response_format, json_schema or strict.
Closest thing today is forcing a tool call, since chat() does take tools and tool_choice:
That guarantees the call and says nothing about the arguments. Anthropic’s real guarantee is strict: true on the tool definition, which constrains sampling to schema-valid tokens, and @forge/llm gives you no way to set it. Keep your validator on the parsed args, and guard the read: tool_calls is optional, and response-mapper.js is a bare await response.json() cast, so nothing enforces the object type on arguments.
My read is that strict is the easier ask than structured outputs. One boolean on the Tool type Atlassian already exposes.
Do you need it enforced, or reliably parseable? If a retry on parse failure is acceptable, the forced call probably covers you.