Hey everyone,
Thanks for all the great comments and responses. I’ll address some of the high level themes, hopefully it catches most:
Consumption and limits
Yes, early adopters of Forge LLMs will need to build user/tenant/edition based limits within their apps. Feature flags will be available by the time LLMs are in preview which could be useful for more dynamic controls, throttling etc.
With Forge pricing generally, you can expect to see a lot more alerts/controls/APIs etc. coming in 2026 that will make it easier to observe and manage use of all Forge paid capabilities. We’re getting lots of good feedback on this, including in the developer space session yesterday.
Expect to see more RFCs etc. on improving the tooling around paid Forge services.
Consumption based pricing on Marketplace
This is still on the Marketplace roadmap, although it has been delayed. I appreciate this will be limiting on the types of use cases you can build for in the short term, but in the medium/long there will be more flexibility around monetisation.
Observability, evals, testing etc.
Obviously we understand how important this is, and something we are doing a lot of internally with Rovo development etc. We won’t have any specific new features at launch but keen to work on better supporting this on Forge.
Some loosely held thoughts:
- I imagine we would lean heavily into offline evaluations (using pre-collected, static datasets before production). So would recommend, if anything, folks start investing in their training data sets.
- Considering that we’re just exposing the vanilla Claude models, for now you could use third party tools to run evals against those models outside of Forge. To test prompt tweaks, different model versions etc.
- As for online evals, expect to have very limited (if any) access to raw prompts/responses in production. We hold ourselves to pretty high standards internally and you can expect that to be reflected in any tooling we build.
- For online you could use things like feature flags (coming soon) and custom metrics to capture click-through rates, thumbs up/down, engagement etc in production. You could also use the RoA supported OTEL/analytics services.
Major versions
Yes @BenRomberg I should have pointed out that this will be a new permission which will be able to be decoupled from versioning as per RFC-106. In that case it wouldn’t be a major version change but the admin would need to accept the new permission before that installation could access the LLM.
SDK suggestions
Lots of good suggestions around things like streaming, prompt caching etc. keep them coming 
We will have some constraints based on Atlassian’s internal AI gateway and the underlying host (AWS Bedrock in the case of these models) but we’ll look into theses.