A model choice that looks cheapest in a benchmark may perform poorly on the tasks an agent actually handles. The more useful signal is the traffic already flowing through production: its successes, failures, and edge cases can inform which model should handle each request and where a specialized model might take over.
What Sets It Apart
- One gateway for different model sources: Hosted providers, users’ own provider keys, and local models can sit behind a shared API. This reduces integration changes when teams switch providers or add fallback options.
- Traffic-informed routing: Traces can be used to simulate workloads and evaluate routing choices. That makes model selection more specific to an application’s real requests than relying only on generic benchmarks.
- Access and spending controls: Teams can assign model permissions and budgets to users or agents, making centralized governance part of the inference layer rather than a separate process.
- A path from routing to model ownership: The project supports optimizing a router and using collected traces to fine-tune an open model. This offers a gradual route toward moving suitable work to a task-specific model while retaining stronger models for harder requests.
Who It’s For—and the Trade-off
Great fit for teams running multiple AI agents or providers that want a common control point, request-level budgets, and a way to learn from their own traces. Look elsewhere if you only need a simple SDK wrapper, do not want to operate a gateway, or cannot collect and evaluate representative production traces. Router quality and any claimed savings depend on the workload and evaluation process; trace-based optimization is not an automatic guarantee of better results.
