Quantisation and surrogate probes

TL;DR: Linear probes are effective at interpreting quantised models. No probe performance is lost down to 4 bits and in fact performance remains strong even beyond that. The linear probes were also shown to outperform baselines such as embedding models.

Motivation
It is known that quantisation can negatively affect a model’s output quality. However, there is evidence that suggests that quantisation does not meaningfully affect the internal activations of the model during the prefill stage, the cheap and efficient part of inference that our probes read from.
We care because Telluvian's work is largely based on surrogate model interpretability. In short, this is when a closed model's transcript is replayed through an open-weight ‘surrogate’ model. Linear probes are then used to detect behaviours in the surrogate that generalise to the original closed weight model. Quantising the surrogate would make this process cheaper and possibly faster. The purpose of this experiment is to test whether quantising the surrogate affects probe performance.
Probe performance was measured across a range of layers and quantisations, on two datasets, against a range of baselines. This builds on our earlier work by Ben Mullins. Ben found evidence to suggest that quantisation costs our probes nothing in terms of performance and that NVFP4 cuts GPU time by around 40%.
Headline result: From 16 down to about 4 bits, no meaningful performance degradation is present. Below that the probes start to measurably degrade.
Method
We used two datasets with different behaviours:
- Coding Agent Misalignment (CAM): 2,318 coding-agent sessions, with tokens labelled where GPT-5.4 flagged the agent as overconfident and lazy. This dataset also included a number of other LLM judges. We elected to use a single model as a judge to provide a consistent behaviour for the probes to learn. GPT-5.4 was chosen as it had the highest positive rate. This decision was practical for our purposes but means the probes have no practical use beyond aping GPT-5.4 reactions to coding agent overconfidence.
- 12,000 question-and-answer records from GPT-5.6-Luna, with positive tokens labelled where GPT-5.6-Terra marked a claim as unsupported.
Both datasets use LLM-judge labels, so the probes primarily learn to imitate a judge rather than detect the behaviour itself. That's not ideal but also not a problem for our purposes. Our primary concern is using a consistent dataset rather than measuring any specific semantics. Full details of the datasets and method are in the appendix.
We first trained probes on the unquantised (bf16) Qwen3.8-27B, as a baseline. We then used each quantised version to answer two questions:
- Reused: do the original probes still work on the quantised model's hidden states? This is what we'd actually deploy.
- Retrained: if we train fresh probes on the quantised states, is the information still there?
We scored both on AUROC and AUPRC at 10 layers. A model counted as equivalent to bf16 if the 95% confidence interval on the difference sat inside ±0.01. This margin was fixed before anything was run.
We also measured the size of the change in each hidden state relative to its original size, labelled as 'drift', where a drift of 0 means no change. Cosine and L2 drift were both used at various points.
We used the following models as part of the study.
| Name | Checkpoint | Bits/weight | Engine |
|---|---|---|---|
| bf16 | Qwen/Qwen3.8-27B @1d4bf0f2 | 16 | vLLM |
| FP8 | Qwen3.8-27B-FP8 @017b9c7a | 8 | vLLM |
| NVFP4 | NVIDIA NVFP4, mixed 4/8 @482ca0f3 | ~4.5 nominal (~6.5 by file size) | vLLM, CUTLASS linear backend |
| Q4_K_XL | unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL | ~4.5 nominal (~5.2 by file size) | llama.cpp |
| Q3_K_XL | same repo, UD-Q3_K_XL | ~3.9 | llama.cpp |
| IQ1_M | same repo, UD-IQ1_M | ~2.0 (not 1-bit, despite the name) | llama.cpp |
| Ternary Bonsai | prism-ml/Ternary-Bonsai-2-27B-gguf @b072e1d3 PTQ1_0 | 1.7 | llama.cpp |
| Bonsai-27B Q1_0 | prism-ml/Bonsai-27B-gguf @f10afb35 (base Qwen3.6-27B @6a9e13bd) | ~1.1 | llama.cpp |
NVFP4 and Q4_K_XL are labelled by nominal format in the figures and are actually higher in practice.
A note on engine control. Both llama.cpp and vLLM were used in this study. As a control, bf16 was run using both llama.cpp and vLLM. It was found that they differ by at most 0.0003 in probe metrics, with hidden-state cosine ≥ 0.995 at every layer. Therefore the differences between models can be confidently attributed to quantisation rather than the engine.
Regarding expectations, quantisation is generally considered lossless and stable down to approximately 4 bits. Therefore we expected this to be a threshold at which things may start to break down. We were unsure how different versions of the same quantisation (i.e. NVFP4 and UD-Q4_K_XL) would perform; our expectation was that they should be essentially the same.
Baselines
We decided to also test probe performance against a number of baselines. The rationale was that if a probe only picked up surface wording, any quantised model would preserve it and the test would tell us nothing useful. So we compared the bf16 probe with text-only baselines: TF-IDF, Qwen3 embeddings, BM25 retrieval, and a model that sees only token position and transcript length. How each baseline works, with full results, is in the appendix.
The linear probe comfortably outperformed the baselines. At layer 49 it beats TF-IDF (the strongest baseline) by 0.19 AUPRC on CAM and 0.26 on Luna. Therefore, the probe appears to be reading something from the model's internal state that can't be gleaned from just the input/output text.

Results
Down to 4 bits
FP8, NVFP4 and Q4_K_XL were equivalent to bf16 at every layer, on both metrics and both datasets, whether we reused the probes or retrained them. The largest delta anywhere was 0.004, which is comfortably within our equivalence margin of 0.01.
| Model | Bits | Prefill speed vs bf16 | Largest change, reused probe (CAM / Luna) |
|---|---|---|---|
| FP8 | 8 | 1.31× | 0.0012 / 0.0012 |
| NVFP4 | ~4.5 | 1.54× | 0.0023 / 0.0036 |
| Q4_K_XL | ~4.5 | 0.89× (llama.cpp) | 0.0019 / 0.0013 |
NVFP4 has the fastest throughput, at approximately 1.5 times bf16 prefill speed on both datasets.

Below 4 bits
Q3_K_XL (~3.9 bits) is marginally smaller than NVFP4 and Q4_K_XL and shows increased degradation as expected. It loses about 0.007 AUPRC at layer 49 on both datasets. It is worth noting that this is still within the equivalence margin. A retrained probe recovers essentially all of this loss.
By around 2 bits the performance loss is more noticeable. IQ1_M's reused probe loses 0.054 AUPRC at layer 49 and falls significantly outside the margin. Retraining helps, but still leaves it about 0.02 down in deeper layers. Bonsai-27B Q1_0 is worse still: the reused probe loses 0.115 AUPRC, and retraining recovers about 85% of that. This result is expected since Bonsai-27B Q1_0 is a 1-bit model.
| Model | Bits | AUPRC change at layer 49, reused (CAM / Luna) | Retrained (CAM / Luna) |
|---|---|---|---|
| Q3_K_XL | ~3.9 | −0.007 / −0.008 | −0.003 / −0.004 |
| IQ1_M | ~2.0 | −0.054 / not run | −0.022 / not run |
| Ternary Bonsai | 1.7 | −0.034 / −0.041 | −0.007 / −0.015 |
| Bonsai-27B Q1_0 | ~1.1 | −0.115 / not run | −0.018 / not run |
Please note that Bonsai-27B Q1_0 is built on Qwen3.6 rather than Qwen3.8, so it is compared with Qwen3.6 instead (see the appendix).
The only model to buck the trend is IQ1_M. It has more bits per weight than ternary Bonsai (2.0 against 1.7) but exhibits more performance loss. This is the case with both reused and retrained probes.
It is worth noting that quantisation performance loss was consistent across both datasets throughout.


Drift
A potential alternative measurement of probe degradation was drift. A probe is just measuring the hidden state and so if the state barely moves, the probe's output shouldn't either. However, as the results show, drift isn't reliable enough to use on its own. Cosine drift is especially poor: it reported 0.996 similarity at a layer where the probe lost AUPRC.

Why do low-bit models degrade?
A further test was done to assess if the performance loss was because low-bit quantisation destroys what the probe reads or if it just changes how/where it is stored. To test this, a linear map was fit from the ternary Bonsai's hidden states back to bf16's. The original bf16 probe was then run on these mapped states. The Luna Hallucinations dataset was used.
| Layer | bf16 | Reused | Per-dimension rescale | Linear map (eval R²) | Linear map recovers | Cosine raw → standardised |
|---|---|---|---|---|---|---|
| 0 | 0.273 | 0.250 | 0.250 | 0.268 (0.94) | 75% | 0.996 → 0.813 |
| 7 | 0.305 | 0.289 | 0.290 | 0.305 (0.81) | 99% | 0.985 → 0.849 |
| 21 | 0.358 | 0.337 | 0.338 | 0.357 (0.80) | 97% | 0.953 → 0.832 |
| 35 | 0.409 | 0.371 | 0.373 | 0.400 (0.77) | 75% | 0.901 → 0.780 |
| 49 | 0.451 | 0.410 | 0.414 | 0.435 (0.78) | 61% | 0.845 → 0.768 |
| 63 | 0.417 | 0.372 | 0.374 | 0.396 (0.76) | 51% | 0.768 → 0.741 |
At layers 7 and 21, the map recovers 97–99% of the lost AUPRC, about as much as retraining a probe. By contrast, rescaling each dimension separately recovers at most 10%. From this, we can conclude that the quantisation is mixing dimensions together, and so a simple rescale can't undo it. Deeper in the model the map recovers less (75% at layer 35, falling to 51% at layer 63), so at least some of the information there is truly lost.

Conclusions
- Down to Q4_K_XL and NVFP4 (~4.5 bits nominal), quantisation has no measurable effect on probe performance at any layer, for reused or retrained probes, on two independent datasets. Every effect is at least 2.5× inside the margin.
- Q3_K_XL (~3.9 bits) is the first format to move a reused probe (L49 AUPRC −0.007 on CAM, −0.008 on Luna). Retraining removes the effect.
- At ~2 bits and below, reused probes degrade significantly and retraining recovers only part of the loss.
- The damage depends on the kind of distortion, not solely on bits or drift.
Limitations
- Only one surrogate family (Qwen) was used.
- There was a very low positive rate for both datasets. As such token rows were sampled with enriched positives.
Next steps
NVFP4 in production. NVFP4 matched bf16 on both datasets while running about 1.5× faster, so the next step is to test it in production. This could raise surrogate throughput and cut serving cost without retraining any probes.
Combining this with other interpretability techniques. We'd like to combine this activation measurement scheme with other interpretability techniques, such as the J-space. This may be particularly useful for improving model selection in Gallery.
