XPerf：让模型路由从“选最强模型”走向“预测谁最适合当前问题”XPerf: From Predicting Model Performance to Better Model Routing

作者：王宇彤 Yutong Wang

研究时间：2026 年 9 月起

状态：持续迭代中

当一个产品可以同时调用多个大语言模型时，“哪个模型最强”其实不是最有用的问题。不同模型在代码、数学、知识问答、长文本理解和 Agent 任务上的优势并不完全相同，而调用成本、延迟和输出长度也各有差异。

因此，更实际的问题是：

面对当前这个具体请求，哪个模型最适合回答它？

XPerf 是 Telluvian 路由系统中的模型表现预测组件。它接收用户的 prompt，并为每个候选模型预测预期表现，例如：

XPerf 并不直接负责整个调用决策。成本、延迟、缓存、输出长度和产品策略仍由外层系统处理；它首先试图把一个更基础的问题做好：同一个 prompt 对不同模型而言，成功的可能性分别有多大？

整个项目也是围绕这个问题逐渐展开的。最初，我们先建立最简单的 router baseline；随后加入 embedding 和语言模型内部 activation；再逐步审计、扩展数据并改进评测方式。随着实验推进，研究问题也从“能否预测模型表现”逐渐变成了更困难的一件事：预测得准之后，为什么仍然不一定选得对？

一个 Router 怎样才算真的有用？

本文中 router 的准确率，指在测试 prompt 中，router 所选模型给出正确答案的比例。模型路由很容易被一个看起来不错的准确率误导。假如一个 router 能达到 85% 的准确率，但始终调用同一个最强模型就能达到 87%，那么这个 router 虽然“会预测”，却没有创造真正的选模价值。

因此，项目最开始并没有急着训练复杂模型，而是先建立一套之后一直沿用的评测框架。其中两项检验泛化能力：在“未见过的 prompt”测试中，同一个 prompt 不会同时出现在训练集和测试集里；在“未见过的 benchmark”测试中，整个 benchmark 都不参与训练。

其中，Best single model 是最重要的现实基线。如果 router 无法稳定超过它，就意味着动态选模暂时还没有比“永远调用一个强模型”更有价值。

Oracle 则是另一个极端。它无法部署，因为它假设我们事先知道每个模型在每道题上的真实得分，但它与最佳单一模型之间的差距（即 headroom）就是路由在该数据集上最多能获得的提升。后续几乎所有 XPerf 实验，都建立在这套评测逻辑上。

从 Embedding Router 到 Activation Probe

第一轮实验采用的是相对简单、容易复现的方法：先用 MiniLM、BGE、MPNet 等现成的 embedding 模型，把每个 prompt 转换成一组概括文本内容的数字（即 embedding），再根据最相似的历史 prompt（k 近邻，kNN）或一个简单的线性模型（Ridge 回归）预测模型表现。

它们是很重要的 baseline，因为如果直接使用现成的 embedding、不做额外训练就已经足够完成路由，那么引入更昂贵的内部 activation 就没有太大必要。

在此基础上，我进一步尝试了 activation probe。Activation probe 是一个小模型，它读取另一个语言模型（我们称之为 proxy 模型）在处理 prompt 时内部计算出的数值，即 activation。背后的假设是：模型内部状态可能不只编码“这道题在说什么”，还包含题目难度、推理结构和其他与最终模型表现相关的信息。

第一次看到明显的 Router 信号

在一个公开的 routing benchmark 上，我保存了每个 prompt 的 activation，并训练了一个所有候选模型共用的小型神经网络，再为每个模型分别输出预测，这就是 shared trunk 设计。

配置在验证集上选出（验证集不参与训练，专门用于选择设置）：读取 proxy 模型的第 52 层，对 prompt 最后 32 个 token 的 activation 取平均，并对三次训练的结果取平均。下表结果来自只用于最终评分的测试集。

这是一个重要的早期结果。至少在相同 benchmark 集合中的未见 prompt 上，activation probe 已经能够超过固定最佳模型，说明内部表征中确实存在可用于模型选择的信号。

但如果只看 64.54% 这个数字，很容易对结果过于乐观。真正的问题出现在更严格的“未见过的 benchmark”测试中。下表中的 BCE（binary cross-entropy）是衡量概率预测误差的指标，越低越好。

训练过程中，训练误差持续下降，但验证误差大约在第三个 epoch（即完整遍历一次训练数据）后开始反弹。与此同时，一旦把整个 benchmark 留到测试阶段，router 的表现便跌到固定模型以下。

这说明模型在熟悉的任务分布中确实学到了有用规律，但其中相当一部分规律可能与 benchmark 本身绑定得太紧。

换句话说，模型可能已经很会判断：

“这种 benchmark 通常哪个模型比较强。”

但距离真正理解：

“这一个具体 prompt 为什么更适合某个模型。”

还有明显差距。

数据比想象中更重要

这一结果让研究重点逐渐从“继续调模型结构”转向了数据本身。

Routing dataset 有一个很容易被忽视的问题：记录数很多，并不意味着问题足够多样。 十万次模型回答可能只是几千个 prompt 的重复采样，而一个 router 真正需要学习的是不同任务、不同能力和不同模型之间的关系。

因此，在继续训练之前，我先对另外两个数据源做了系统审计：第二个 routing dataset，以及一个大型 benchmark 结果数据库。

第二个 Routing Dataset：先确认这里有没有值得路由的空间

该数据集共包含 43,288 条记录和 14 个候选模型。完成数据检查，并重新划分数据、确保同一个 prompt 不会同时出现在训练集和测试集之后，测试集上出现了一个非常重要的结果：

Oracle 比最佳固定模型高出 14.10 个百分点，说明这个数据集里确实存在明显的模型选择空间。与此同时，官方划分中有 108 个 prompt 同时出现在训练集和测试集中，这也是重新划分数据的原因。

目前该数据集已经完成审计与训练准备，但尚未提取 activation，也尚未训练新的 probe。

Benchmark 数据库：64 万次回答，其实只有 3,340 个问题

这个 benchmark 数据库的规模看起来更大：

这里最值得注意的是第一行和第二行之间的差异。642,752 次 attempts 实际只对应 3,340 个独立 prompt，因此不能简单把几十万条回答当成几十万种不同的训练问题。

重复 attempts 仍然很有价值，它们可以帮助估计同一个模型在同一个问题上的表现分布和不确定性；但它们不能替代 benchmark 和任务类型上的真正多样性。

审计中还确认了几个数据处理细节：N grade 代表不得分，而不是缺失值；361 条未评分的尝试需要排除；66.42% 的 prompt-模型组合只有一次评分；此外，在一个代码 benchmark 中，两个编程 agent 的结果被标错了。两个数据源之间没有发现完全相同的 prompt。

这部分工作虽然没有直接产生新的 router accuracy，却为后续训练建立了更可靠的数据基础。

升级 Routing Dataset：从预测表现到跨模型 Calibration

新的 routing dataset 到位后，我在 GPU 上为它提取了 activation，并首先只用 10 个候选模型都有结果的 prompt 进行训练。

只使用 3,904 个完整覆盖的 prompt，是为了确保所有候选模型拥有相同的训练机会，避免缺失标签本身影响模型比较。训练、选层与最终测试也严格分离：训练集用于拟合，验证集用于选择层，测试集只在最后报告。

最终在验证集上选择了第 44 层。下表中的 AP（average precision，平均精度）衡量预测能否把某个模型的成功排在失败之前，1.0 为完美。

这一轮实验出现了一个非常关键的矛盾。

平均 per-model AP 已经达到约 0.949，远高于仅凭基础比例得到的 0.822。这说明 probe 对“某个模型会不会成功”拥有很强的预测能力。

但最终 router 的准确率只有 86.22%，仍略低于固定使用 GPT-6 Astra 的 86.87%。这与最强模型相差不到 1 个百分点，而每次请求的成本约为其一半，因为 router 只把 7.5% 的 prompt 交给了 Astra：

能够分别预测每个模型的表现，并不意味着这些预测已经能够被直接横向比较。

这也第一次把 XPerf 的核心问题明确地指向 cross-model calibration，即某个模型的预测值 0.8 与另一个模型的 0.8 是否代表同样的含义。不同模型的预测概率可能各自很准确，但它们的尺度未必完全一致，因此简单取最大值并不一定会选出最合适的模型。

Shared Trunk：不同模型之间确实存在共享结构

随后，我在数据集的一个中间版本上，比较了“每个模型单独一个网络”和 shared trunk 设计。

Shared trunk 比单独网络提高了 1.71 个百分点，说明不同候选模型的表现预测之间存在可以共享的结构，而不是 17 个完全互不相关的问题。

不过，91.43% 仍没有超过 92.57% 的最佳固定模型。也就是说，共享表征让 prediction 变得更好，但“更好的 prediction”依然没有完全转化成“更好的 routing”。

这一阶段也让工作更接近部署：我把 router 连同其设置和诊断结果一起打包，使产品侧能够加载、复现和检查它。

把“表现”和“成本”拆开：XPerf、XLength 与 XCost

随着研究继续推进，系统边界也变得更加清晰。XPerf 最适合专注于预测模型表现，而成本和输出长度应该作为独立信号处理，而不是全部塞进一个 router 中。

因此，后续增加了 XCost 与 XLength。XLength 预测模型对当前请求的回复会有多长（以 token 计），为外层成本估计提供输入；XCost 则用于比较不同路由策略与简单 prompt-length heuristic 的质量—成本关系。

整个架构可以简单表示为：

XPerf   → 预期回答质量
XLength → 预期输出长度

        ↓

外层策略 → 综合质量、成本、延迟与缓存做最终选择

这种拆分的好处是职责清晰。XPerf 不需要自己预测最终账单，XLength 也不需要判断答案质量；产品层可以根据不同使用场景决定如何权衡这些信号。

Activation 的优势，只是因为它“看得更长”吗？

早期 embedding baseline 有一个很明显的潜在不公平因素：MPNet 的输入长度大约只有 384 tokens，而 activation probe 可以看到更长的 prompt。

因此，activation 表现更好可能有两种解释：一种是 activation 本身真的包含更多与模型表现有关的信息；另一种只是 activation 看到了更多文本。

最后这组对比专门用来区分这两个解释。

这组对比使用了一个更大的数据集，包含 11 个 benchmark 中的 7,313 个 prompt 和 17 个候选模型，并有固定的训练、验证和测试集。Embedding baseline 换成了 Qwen3-Embedding-8B，它一次可以读取约 32,000 个 token，因此能够在不截断的情况下编码全部 7,313 个 prompt。其余训练设置保持一致。

最终对比如下：

Activation probe 在 11 个 benchmark 中的 10 个上取得更低 BCE。

这说明此前看到的优势并不能简单归因于 context window。即便 embedding model 能完整看到长 prompt，proxy 模型（Qwen3.8-27B）第 50 层的 activation 仍然提供了更强的模型表现预测信号。

从目前的实验看，普通 embedding 很擅长描述“prompt 是什么”，而语言模型内部 activation 似乎还能保留一些与“模型面对这个 prompt 会表现怎样”有关的额外结构。

Prediction 已经很强，Router 到底还缺什么？

走到这里，XPerf 已经可以相当不错地预测不同模型的表现，但研究中开始出现一个越来越明显的可能性：模型主要学到了两类信号。

第一类是 Prompt Difficulty——这道题整体有多难；第二类是 Global Model Quality——某个模型平均而言有多强。

这两种信息足以产生非常好的 per-model prediction。例如，当一个非常困难的问题出现时，所有模型的成功概率可能同时降低；简单问题则可能让所有模型的预测一起升高。

问题在于，router 真正需要的不只是这些共同变化。

它还需要知道：

在这一个具体 prompt 上，哪个模型会相对于自己的平均水平表现得更好？

这就是 prompt-specific residual。

一个模型可能整体比另一个模型强，但在代码、数学、事实检索或特定推理结构上，相对优势可能发生变化。真正值得路由的部分，恰恰来自这种 prompt-specific 的相对差异。

下一步：把“预测得准”变成“选得对”

接下来的实验会直接检验上面的假设。

首先是 Residual Target。与其让模型直接预测每个候选模型的原始成功率，可以改为预测它相对于当前 prompt 平均表现的偏差。这样做的目的，是先削弱所有模型共同拥有的 difficulty signal，让模型把更多容量用于学习候选模型之间真正不同的部分。

另一个重要实验是建立一个非常简单、可解释的 Additive Baseline：

它是需要首先排除的最简单解释：也许根本不需要学习复杂的“模型—问题匹配关系”，只要知道题目有多难，再知道每个模型平均有多强，就已经可以解释大部分预测结果。

如果复杂的 activation probe 与这个简单 baseline 表现接近，就说明当前模型主要学习到的仍然是共享 difficulty 和 global model quality。只有当 residual model 能稳定超过这个 baseline，才更有证据说明模型开始捕捉真正的 prompt-specific routing signal。

后续还会测试直接针对路由选择打分的训练目标，例如让模型两两比较排序，以及最小化 regret（相对于最佳选择所损失的准确率），并通过“未见过的 benchmark”测试观察这些信号在新任务上的迁移能力。这里的目标不再只是继续把 BCE 降低一点，而是让训练目标更直接地服务于最终路由决策。

为什么数据多样性可能比模型结构更重要？

目前几个阶段的实验共同指向了同一个问题：router 的瓶颈很可能不只是模型结构，而是训练数据本身。

如果训练数据只覆盖少数 benchmark，一个很强的 probe 完全可能学会识别 benchmark，然后记住“这个 benchmark 通常哪个模型表现最好”。这种策略在新 prompt 来自熟悉 benchmark 时可以获得不错结果，却很容易在“未见过的 benchmark”测试中失效。

因此，下一阶段的数据扩展不会单纯追求更多 attempts，而会更关注 benchmark 和任务类型的多样性、frontier model coverage、记录每条结果的来源、重复测量以及能够支持“未见过的 benchmark”测试的数据结构。所有这些数据源也会逐步标准化到统一的 prompt / model / score 格式，同时保留来源和置信度等信息。

这也是为什么前面的数据审计并不是项目中的辅助工作，而是 router 能否真正泛化的核心部分。

从 Research Probe 到真正可用的 Router

回顾整个项目，XPerf 最初的问题其实很简单：

我们能不能根据一个 prompt 预测不同模型的表现？

目前的实验已经说明，这件事是可以做到的，而且 activation probe 能够提供相当强的预测信号。与此同时，我们也逐步建立起了 embedding baseline、activation 提取、shared trunk 训练、更严格的评测方式、数据审计、XLength 以及可部署的 router。

但真正接近产品的问题比最初的 prediction task 更难：

如何把对模型表现的预测，转化成稳定、可泛化、真正优于固定模型的选模能力？

这意味着下一阶段的重点不会只是继续优化一个单独的 loss，也不会只是让 per-model AP 再高一点。更重要的是确认 XPerf 是否真正捕捉到了不同模型面对同一个 prompt 时的相对优势，并且这种优势能否迁移到训练时没有见过的新任务上。

最终，XPerf 想回答的不是“哪个模型总体最强”。而是：

面对这一次具体请求，哪个模型最适合回答它？

Author: Yutong Wang

Research period: September 2026 onwards

Status: Ongoing

When a product can call several large language models, "which model is strongest?" is the wrong place to start. Models differ in how well they handle code, maths, knowledge questions, long documents and agent tasks, and they differ in price, speed and how much they write. What matters is which model is most likely to answer this particular request well.

XPerf is the part of Telluvian's router that estimates this. Given a prompt, it predicts how each candidate model will perform, for example:

XPerf doesn't make the final call. Cost, latency, caching and product preferences are handled by the rest of the system, and XPerf answers a narrower question: for this prompt, how likely is each candidate to do well?

The project started with simple routing baselines, moved on to text embeddings and then to the internal activations of a language model, and later turned to auditing datasets, building larger ones and testing more strictly. Along the way the question changed. Predicting model performance turned out to be achievable. The harder problem was why good predictions didn't always lead to better choices.

Establishing a Reliable Routing Baseline

Throughout this post, a router's accuracy is the share of test prompts on which the model it picked gave a correct answer. A high accuracy can still be worthless. A router that scores 85% adds nothing if always picking the same strong model scores 87%.

So before training anything complicated, I set up the baselines that every later experiment is measured against. Two of them check whether a router generalises. In the unseen-prompts test, no prompt appears in both the training and test data. In the unseen-benchmarks test, a whole benchmark is left out of training.

The best single model is the comparison that matters most in practice. Until a router beats it consistently, there's no case for routing over calling one strong model every time.

The oracle is the other extreme. It can't be deployed, because it assumes we already know how every model did on every prompt. But the gap between the oracle and the best single model, the headroom, is the most routing could ever gain on a given dataset.

From Embedding Routers to Activation Probes

The first experiments used simple methods that are easy to reproduce. Each prompt was turned into an embedding, a list of numbers summarising what the text is about, using off-the-shelf models such as MiniLM, BGE and MPNet. Performance was then predicted from the most similar past prompts (k-nearest neighbours) or with a simple linear model (ridge regression).

If an off-the-shelf embedding were good enough for routing, there would be little reason to build anything more expensive, so these methods set the bar.

Next I tested an activation probe: a small model trained on the activations of a separate language model, which we call the proxy. Activations are the internal numbers a model computes as it reads a prompt. My hypothesis was that they reflect how hard a prompt is and what kind of reasoning it needs, as well as its topic.

The First Clear Routing Signal

On a public routing benchmark, with activations saved for every prompt, I trained a small neural network shared by all candidate models, with a separate output for each. This is the "shared trunk" design.

I chose the settings on a validation set held back for that purpose. The best configuration read layer 52 of the proxy, averaged its activations over the last 32 tokens of the prompt, and averaged three training runs. The results below come from the test set, which was used only for the final score.

On new prompts from benchmarks it had seen in training, the activation-based router beat the best fixed model by almost six points. That was the first sign that the activations held information useful for choosing a model.

The unseen-benchmarks test was less encouraging. The table below also reports BCE (binary cross-entropy), an error score for probability predictions where lower is better, and how much of the available headroom the router captured.

Training error kept falling, but validation error began to rise after about the third epoch (a full pass through the training data). And once a whole benchmark was held out, the router did worse than the fixed model.

My reading is that the model had learnt real patterns, but many of them were tied to particular benchmarks. It had learnt something like "this kind of model does well on this benchmark", when what we want is "this prompt suits this model". Much of the later work is about closing that gap.

Data Matters More Than It First Appears

These results pointed me away from tuning the model and towards the training data.

A routing dataset can have a huge number of records and still not much variety. Hundreds of thousands of responses may come from only a few thousand distinct prompts, and a router needs to see many different kinds of task to learn how models differ.

Before training anything else, I audited two more data sources: a second routing dataset and a large database of benchmark results.

A Second Routing Dataset: Is There Room to Gain?

This dataset has 43,288 records across 14 candidate models. After checking data quality and rebuilding the split so that no prompt appears in both training and test data, the test set showed a large gap between the best fixed model and the oracle.

A 14.10-point gap means there is plenty to gain from choosing models well on this data.

The audit also found 108 prompts that appeared in both the official training and test splits, which is why I rebuilt the split.

The dataset is now ready for training, but I haven't yet extracted activations for it or trained a new probe.

A Benchmark Database: 642,752 Attempts, but Only 3,340 Distinct Prompts

The benchmark database looks much bigger:

Compare the first two rows. 642,752 attempts sounds like a lot of training data, but they cover only 3,340 distinct prompts.

Repeated attempts aren't wasted, since they show how much a model's performance on the same prompt varies. They can't make up for a narrow range of benchmarks and tasks, though.

The audit also sorted out some labelling problems. An N grade means no credit, not a missing label. The 361 ungraded attempts need to be excluded, and 66.42% of prompt-model pairs have only one scored attempt. In one coding benchmark, results from two coding agents had been mislabelled. There was no exact prompt overlap between the two sources.

None of this raised the router's score directly, but the later training depends on it.

Expanding the Routing Dataset: From Performance Prediction to Cross-Model Calibration

When the newer routing dataset arrived, I extracted activations for it on GPU and started by training only on prompts that had results for all 10 candidate models.

Using only the 3,904 fully covered prompts gave every model the same training data, so missing labels couldn't skew the comparison. As before, the training set was used for fitting, the validation set for choosing the layer, and the test set only for the final report.

Validation picked layer 44. The table also reports average precision (AP), which measures how well the predictions rank a model's successes above its failures; 1.0 is perfect.

This is where the central tension of the project showed up. Mean per-model AP was about 0.949, well above the 0.822 you would get from the base rate alone, so the probe was very good at predicting whether each model would succeed.

The router itself reached 86.22%, slightly below the 86.87% from always using GPT-6 Astra. That is within a point of the strongest model at about half the cost per query, because the router sent only 7.5% of prompts to Astra:

Predicting each model well, one at a time, doesn't guarantee that the predictions can be compared with each other. This is the problem of cross-model calibration: whether a predicted 0.8 for one model means the same as a predicted 0.8 for another. Each estimate can be accurate on its own while the scales differ slightly, and then picking the highest prediction won't always pick the best model.

Shared Trunk: Learning Structure Across Candidate Models

Next, on an interim version of the dataset, I compared a separate network for each model with the shared-trunk design.

The shared trunk was 1.71 points more accurate than separate networks, which suggests the 17 prediction problems have a good deal in common.

It still fell short of the 92.57% from always using the best fixed model, so again, better predictions didn't fully turn into better routing.

I also packaged this version of the router with its settings and diagnostics so the product team could load, reproduce and inspect it.

Separating Performance from Cost: XPerf, XLength and XCost

As the system took shape, it made sense to keep XPerf focused on quality and to handle cost and reply length as separate signals.

That led to two more components. XLength predicts how long a model's reply to a request will be, in tokens, which feeds into the cost estimate. XCost compares the quality and cost of different routing strategies against simpler rules, such as routing on prompt length.

The design looks like this:

XPerf   → Expected answer quality
XLength → Expected output length

              ↓

Outer policy → Final decision across quality, cost,
               latency and caching

Each component has one job. XPerf doesn't need to estimate the bill, and XLength doesn't need to judge answer quality. The product layer decides how much weight each signal gets for a given use case.

Testing the Value of Activations with Long-Context Embeddings

The earlier embedding baseline had a handicap. MPNet only reads about 384 tokens, while the activation probe saw much longer prompts. So the activations might have done better because they carry more useful information, or simply because they saw more of the prompt.

I set up a final comparison to tell these apart. It used a larger dataset of 7,313 prompts across 11 benchmarks and 17 candidate models, with fixed training, validation and test sets. The embedding baseline became Qwen3-Embedding-8B, which reads about 32,000 tokens at once and so could take in all 7,313 prompts without truncation. Training was otherwise unchanged.

The activation probe had lower error on 10 of the 11 benchmarks.

Context length doesn't explain the gap, then. Even when the embedding model sees the whole prompt, activations from layer 50 of the proxy model (Qwen3.8-27B) predict model performance better. On these experiments, embeddings seem to capture what a prompt is about, while activations also carry information about how models are likely to do on it.

Current Diagnosis: Difficulty, Model Quality and Prompt-Specific Residuals

By now XPerf predicted individual model performance well. Much of that predictive power, though, may come from two fairly simple signals: prompt difficulty, how hard the request is overall, and global model quality, how good a model is on average.

Those two alone can produce strong per-model predictions. A hard prompt pulls every model's predicted success down, and an easy one pushes them all up.

A router also needs to know whether a particular model will do better or worse than usual on this particular prompt. I call this the prompt-specific residual. A model that is stronger overall can still lose its edge on certain kinds of coding, maths or factual questions, and those prompt-by-prompt differences are where routing should get most of its value.

Next Step: Turning Predictive Performance into Routing Advantage

The next experiments test this directly.

The first is a residual target. Rather than predicting each model's raw chance of success, the probe predicts how far each model sits from the average on that prompt. This strips out the difficulty signal that all models share and makes the probe concentrate on where they differ.

The second is a deliberately simple additive baseline:

It is the simplest explanation to rule out first. Perhaps there's no need to learn how prompts and models interact, and knowing how hard the prompt is and how good each model is already accounts for most of the prediction.

If the activation probe does no better than this baseline, most of its signal comes from shared difficulty and model quality. If a residual model beats it consistently, that is stronger evidence that the probe is picking up genuine prompt-specific signal.

I also plan to try training objectives that score the routing choice directly, such as ranking models against each other in pairs and minimising regret (the accuracy lost compared with the best possible choice), each measured on the unseen-benchmarks test. At this point, a training objective closer to the routing decision is worth more than a slightly lower BCE.

Data Diversity and Cross-Task Generalisation

Several parts of this work point the same way: the bottleneck may be the variety of the training data more than the model.

If the training data covers only a few benchmarks, a strong probe can learn to recognise each benchmark and remember which model usually wins there. That works on new prompts from familiar benchmarks and fails on a benchmark it has never seen.

The next round of data collection is about breadth rather than volume. That means more benchmarks and task types, better coverage of frontier models, a record of where each result came from, keeping repeated measurements, and datasets built so that the unseen-benchmarks test means something.

I'll also convert all the data sources into one prompt / model / score format that keeps source and confidence information. Whether the router generalises depends heavily on this auditing work.

From Research Probe to Deployable Router

XPerf started with a simple question: can we predict how different models will perform from the prompt alone? The experiments so far say yes, and activations are a strong signal for it.

The harder question is how to turn those predictions into model choices that reliably beat calling one fixed model, including on tasks the router never saw in training. That depends on whether XPerf can learn where models differ on the same prompt, which is what the residual and data work above sets out to test.