Are you considering to allow control over the model selection down the road (example)?
Our use cases require as deterministic as possible responses, so despite the ingrained non-deterministic aspects, it seems to be a considerable business risk to built an app atop a model that might change under the hood and yield substantially different results all of a sudden.
Short answer, yes. Even as we were launching in Barcelona, PMs were telling me they’ve been thinking in that direction. We discussed how smaller LLMs are cheaper and faster, while larger are more expensive and slower. Even for that simple consideration, “one size fits all” is a poor model (given how expensive and slow the newest/largest models are).
Even as I see the value in our strategy to treat models as “hidden commodity services” (like we do with other resources like compute and storage), I worry about the non-determinism too. Do you think the only way to mitigate that is precise coupling to models?