Question about LLM Tracing and Observability for Forge LLM

Forge
1

Hi,

We are currently working with Forge and exploring the capabilities of
the Forge LLM. As part of our development process, we are very
interested in understanding whether Atlassian is planning to provide
an official framework or tooling for analyzing and observing LLM
behavior.
More specifically, we would like to know:
* Whether there are plans to provide tracing/observability
capabilities for Forge LLM calls.
* Whether developers will be able to inspect execution traces (e.g.,
prompt flow, tool calls, intermediate steps, outputs, etc.).
* Whether Atlassian is considering integration with external
observability platforms such as LangSmith (or similar tools).
* If such functionality is planned, is there an approximate ETA or
roadmap visibility for when it might become available?
For us, visibility into LLM behavior (including tool usage and
intermediate reasoning steps where applicable) is critical for
debugging, evaluation, and improving reliability in production
environments.
Any guidance on current capabilities, planned features, or recommended
best practices for LLM observability within Forge would be highly
appreciated.
Thank you in advance for your help.

2

Hi @OlehDanylevych,

Thanks for the question.

To maintain this high bar we set for Runs on Atlassian, our general principle is to not expose raw production prompts and responses. We believe this is important for winning the trust of enterprise customers. So this means there will always be limitations on what you can do with traces - specifically in production.

So at the moment our thinking is

1. Lean into to Offline Evals (using tools external to Forge)

For debugging prompt flow, tool calls, and reasoning steps, we encourage a test-Driven approach using offline evaluations.

  • Build a robust set of “Golden Test Cases” that represent your app’s expected behavior.

  • Leverage third party tools/frameworks like LangSmith, Braintrust, or promptfoo outside of the Forge environment during development. This allows you to iterate on your prompt engineering and tool-call logic using synthetic or de-identified data before deploying to Forge.

2. Lean into “Behavioural” Observability

Instead of looking at the raw traces, create visibility into the execution of the LLM. Some ideas you can already try and others we’re thinking about:

  • Detailed Metadata in API Responses: We already provide metadata around finish_reason and token usage - but potentially there is other metadata we could provide that would help you understand success?

  • Custom Metrics: Custom metrics are currently in EAP but should be available for prod un the coming quarter. We suggest logging high-level outcome metrics (e.g., llm_success_rate, tool_call_retry_count) or user feedback (thumbs up/down) to pinpoint production issues without needing the raw logs. These can be exported to a tool of choice when customers allowed.

3. Roadmap

While we don’t have an ETA for a “LangSmith-style” native integration today, it’s not something we’re actively planning. It is possible that we might allow something like that in dev environments only. It really depends on the adoption of Forge LLM and whether we prioritise it over other areas like containers and SQL, which we’d also like to improve the observability of.

Anyway, thanks for kicking off the conversation. It’s obviously early days for Forge LLMs so it’s all super valuable.