Routing Across the Model Frontier

"What is the capital of Peru?" and "Why does this function deadlock under high load?" are very different requests, yet most applications approach these problems with the same AI tool. Of course, almost any model can answer the first for a fraction of a cent. The second prompt may need the best model available.
Model routing picks a model for each request as it arrives. This post explains how we do it at Telluvian, by predicting how well each model will answer and what it will cost before any of them runs.
Why one default model isn't enough
Dozens of models are now on offer from providers such as OpenAI, Anthropic and Google, at very different prices and with different strengths. Sending everything to the strongest model means overpaying for routine work like sorting support tickets, while sending everything to a cheap one gives up capability on the requests that need it.
A router sends simple requests to cheap, fast models and hard ones to frontier models. Frontier models meaning the latest and greatest in AI models available today. Using a router also means that if one provider is down, another can take over. By way of example, imagine choosing between three models like these. Expected performance is the predicted chance that the model will correctly answer a given prompt:
| Candidate model | Expected performance | Relative cost | Typical use |
|---|---|---|---|
| Efficient model | 76% | Low | Routine, high-volume work |
| Balanced model | 84% | Medium | General-purpose requests |
| Frontier model | 89% | High | Difficult or quality-sensitive tasks |
In this case, the balanced model is nearly as likely to get it right as the frontier model, for less. A user may prioritise performance or cost and that preference will inform the model choice. This sort of smart model selection is what sets a router apart from just reading a leaderboard that ranks models by their averages.
The economics
Providers charge per token, a chunk of text about three-quarters of a word long, with separate prices for what you send and what comes back. For a request with 2,000 input tokens and 500 output tokens, at OpenAI's list prices:
| Model | Input price / 1M tokens | Output price / 1M tokens | Approx. request cost |
|---|---|---|---|
| GPT-5 nano | $0.05 | $0.40 | ~$0.0003 |
| GPT-5 | $1.25 | $10.00 | ~$0.0075 |
That's a 25× difference within one model family, or roughly $300 against $7,500 over a million requests. Both will tell you the capital of Peru. However, on a hard reasoning problem, the better answer will be worth the price.
The technical challenge: predicting before sending the call
To be effective, the router has to choose before any model has answered. In theory, we could run every candidate and then just keep the best response to make the choice trivial, but you would have paid for all of them and waited for the slowest response.
So for each prompt and model, we estimate performance and cost in advance. It helps to split that performance estimate into three parts:
Predicted performance ≈ Prompt difficulty + Global model quality + Prompt-specific model fit
Difficulty affects every model and global quality is how good a model is on average. Prompt-specific fit is more of a fine-tuning term. A model that is weaker overall can still be unusually good at SQL or a certain kind of maths problem (look up benchmaxxing if curious).
At Telluvian we make two estimates: XPerf, how likely each model is to answer the next turn well, and XCost, what that turn would cost.
XPerf
XPerf estimates, for each candidate model, the probability that it will successfully answer the next request in a conversation. Therefore, XPerf reads the whole conversation, earlier turns included. Cost plays no part in XPerf; that's left to the routing policy.
Why activations, not embeddings
The obvious input for a predictor is an embedding, a list of numbers that summarises what a text is about. Embeddings are cheap, and prompts on similar topics get similar ones.
Activations are the numbers a language model computes internally as it reads. We run each prompt through a separate proxy model and give its activations to XPerf. Because they come from a model working through the request, they literally reflect how hard it is as well as what it is about.
Against a long-context embedding model given the same prompts in full, activations predicted performance better on 10 of the 11 benchmarks (details in the research log). Our working explanation is that they carry information about how a language model will cope with a prompt, which is what a router needs.
XCost
In a conversation, cost depends on more than list price. With prompt caching, providers charge less for text a model has recently processed, and once a conversation has moved between models, each has cached a different amount of it.
Suppose Model A answered the first turn, then Model B received the conversation so far and answered the second. When the third message arrives:

| Candidate model | Cached context by Turn 3 | Additional context to process | Predicted success | Expected serving cost |
|---|---|---|---|---|
| Model A | Q1 + A1 | Q2 + A2 + Q3 | 74% | Medium |
| Model B | Q1 + A1 + Q2 + A2 | Q3 | 78% | Low |
| Model C | None | Full conversation | 95% | High |
Each input token is billed once: at the discounted rate if the model has it cached, at the full rate if not. XCost adds those two charges to the cost of the reply:
Expected cost = (uncached tokens × input price)
+ (cached tokens × cached input price)
+ (expected reply tokens × output price)
Prices are fixed, we know how long the conversation is and what each model has cached, so the only unknown is the length of the reply. That's also something we predict.
Model B is cheapest, with most of the conversation already cached. Model C is far more likely to answer well but starts from scratch. A warm cache is a reason to stay put, but it shouldn't keep a conversation hostage.
Turning predictions into a decision
What to do with the two predictions is a question of policy.
| Model | XPerf | XCost |
|---|---|---|
| A | 87% | Low |
| B | 91% | High |
A high-volume consumer app might pick A, and a research tool might pick B. Alternatively, other factors may influence the decision. For example, an app that needs fast answers might choose whichever of A and B has the smallest latency. In broad strokes then we can think of the routing policy as:
Utility(model) = XPerf − Cost penalty − Latency penalty − Reliability penalty + Product preferences
How much each term counts is a product decision. This allows us to improve the basic prediction model without assuming anyone's commercial priorities.
Where we go next
Routing for a specific workload. A legal assistant and a coding agent send very different requests and care about different things. Given examples from one application, a router can learn which models are unusually strong or cheap for that workload, rather than relying on one global ranking.
Latency as part of the request. Live support needs an answer straight away, but background research can usually wait, and a request that can wait can be batched or sent to cheaper capacity. Sail Research is built on that idea, and a router could make the same call for each request.
Routing as a layer in the AI stack
As new models arrive, a router can treat each one as another option to weigh, so the application doesn't have to be rebuilt around it. At Telluvian we're building that as a gateway that sends each request to the model that fits it.
