All posts
Adam Pattenden

Quantisation and surrogate probes

The Telluvian brace mark: the left brace drawn smooth, the right brace broken into square pixels that grow coarser from top to bottom over a faint trace of its true curve.

TL;DR: Linear probes are effective at interpreting quantised models. No probe performance is lost down to 4 bits and in fact performance remains strong even beyond that. The linear probes were also shown to outperform baselines such as embedding models.

AUROC change against the 16-bit unquantised model for each quantised model with the original probe reused, CAM in blue and Luna in orange. FP8, NVFP4, Q4_K_XL and Q3_K_XL sit close to zero, well inside the ±0.01 equivalence margin. Ternary Bonsai at 1.7 bits falls to around −0.01 on both datasets.
AUROC change against the 16-bit unquantised model, using the original probe unchanged. Models down to about 4 bits stay well inside the ±0.01 margin on both datasets.

Motivation

It is known that quantisation can negatively affect a model’s output quality. However, there is evidence that suggests that quantisation does not meaningfully affect the internal activations of the model during the prefill stage, the cheap and efficient part of inference that our probes read from.

We care because Telluvian's work is largely based on surrogate model interpretability. In short, this is when a closed model's transcript is replayed through an open-weight ‘surrogate’ model. Linear probes are then used to detect behaviours in the surrogate that generalise to the original closed weight model. Quantising the surrogate would make this process cheaper and possibly faster. The purpose of this experiment is to test whether quantising the surrogate affects probe performance.

Probe performance was measured across a range of layers and quantisations, on two datasets, against a range of baselines. This builds on our earlier work by Ben Mullins. Ben found evidence to suggest that quantisation costs our probes nothing in terms of performance and that NVFP4 cuts GPU time by around 40%.

Headline result: From 16 down to about 4 bits, no meaningful performance degradation is present. Below that the probes start to measurably degrade.

Method

We used two datasets with different behaviours:

  • Coding Agent Misalignment (CAM): 2,318 coding-agent sessions, with tokens labelled where GPT-5.4 flagged the agent as overconfident and lazy. This dataset also included a number of other LLM judges. We elected to use a single model as a judge to provide a consistent behaviour for the probes to learn. GPT-5.4 was chosen as it had the highest positive rate. This decision was practical for our purposes but means the probes have no practical use beyond aping GPT-5.4 reactions to coding agent overconfidence.
  • 12,000 question-and-answer records from GPT-5.6-Luna, with positive tokens labelled where GPT-5.6-Terra marked a claim as unsupported.

Both datasets use LLM-judge labels, so the probes primarily learn to imitate a judge rather than detect the behaviour itself. That's not ideal but also not a problem for our purposes. Our primary concern is using a consistent dataset rather than measuring any specific semantics. Full details of the datasets and method are in the appendix.

We first trained probes on the unquantised (bf16) Qwen3.8-27B, as a baseline. We then used each quantised version to answer two questions:

  • Reused: do the original probes still work on the quantised model's hidden states? This is what we'd actually deploy.
  • Retrained: if we train fresh probes on the quantised states, is the information still there?

We scored both on AUROC and AUPRC at 10 layers. A model counted as equivalent to bf16 if the 95% confidence interval on the difference sat inside ±0.01. This margin was fixed before anything was run.

We also measured the size of the change in each hidden state relative to its original size, labelled as 'drift', where a drift of 0 means no change. Cosine and L2 drift were both used at various points.

We used the following models as part of the study.

NameCheckpointBits/weightEngine
bf16Qwen/Qwen3.8-27B @1d4bf0f216vLLM
FP8Qwen3.8-27B-FP8 @017b9c7a8vLLM
NVFP4NVIDIA NVFP4, mixed 4/8 @482ca0f3~4.5 nominal (~6.5 by file size)vLLM, CUTLASS linear backend
Q4_K_XLunsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL~4.5 nominal (~5.2 by file size)llama.cpp
Q3_K_XLsame repo, UD-Q3_K_XL~3.9llama.cpp
IQ1_Msame repo, UD-IQ1_M~2.0 (not 1-bit, despite the name)llama.cpp
Ternary Bonsaiprism-ml/Ternary-Bonsai-2-27B-gguf @b072e1d3 PTQ1_01.7llama.cpp
Bonsai-27B Q1_0prism-ml/Bonsai-27B-gguf @f10afb35 (base Qwen3.6-27B @6a9e13bd)~1.1llama.cpp

NVFP4 and Q4_K_XL are labelled by nominal format in the figures and are actually higher in practice.

A note on engine control. Both llama.cpp and vLLM were used in this study. As a control, bf16 was run using both llama.cpp and vLLM. It was found that they differ by at most 0.0003 in probe metrics, with hidden-state cosine ≥ 0.995 at every layer. Therefore the differences between models can be confidently attributed to quantisation rather than the engine.

Regarding expectations, quantisation is generally considered lossless and stable down to approximately 4 bits. Therefore we expected this to be a threshold at which things may start to break down. We were unsure how different versions of the same quantisation (i.e. NVFP4 and UD-Q4_K_XL) would perform; our expectation was that they should be essentially the same.

Baselines

We decided to also test probe performance against a number of baselines. The rationale was that if a probe only picked up surface wording, any quantised model would preserve it and the test would tell us nothing useful. So we compared the bf16 probe with text-only baselines: TF-IDF, Qwen3 embeddings, BM25 retrieval, and a model that sees only token position and transcript length. How each baseline works, with full results, is in the appendix.

The linear probe comfortably outperformed the baselines. At layer 49 it beats TF-IDF (the strongest baseline) by 0.19 AUPRC on CAM and 0.26 on Luna. Therefore, the probe appears to be reading something from the model's internal state that can't be gleaned from just the input/output text.

Five panels of bf16 probe metrics by layer on CAM: AUROC, AUPRC, lift over base rate, precision at 50% recall and best F1. Each rises with depth to a peak at layer 49, with dashed lines marking the best baseline of each type below the probe from layer 14 onwards.
The bf16 probe on CAM across layers, compared with the best baseline of each type (dashed). The probe pulls ahead from layer 14 and peaks at layer 49.

Results

Down to 4 bits

FP8, NVFP4 and Q4_K_XL were equivalent to bf16 at every layer, on both metrics and both datasets, whether we reused the probes or retrained them. The largest delta anywhere was 0.004, which is comfortably within our equivalence margin of 0.01.

ModelBitsPrefill speed vs bf16Largest change, reused probe (CAM / Luna)
FP881.31×0.0012 / 0.0012
NVFP4~4.51.54×0.0023 / 0.0036
Q4_K_XL~4.50.89× (llama.cpp)0.0019 / 0.0013

NVFP4 has the fastest throughput, at approximately 1.5 times bf16 prefill speed on both datasets.

Absolute AUROC and AUPRC at layer 49 for each quantised model on CAM, for reused and retrained probes, against a vertical line at bf16. FP8, NVFP4 and Q4_K_XL sit on the line; Q3_K_XL is slightly left; IQ1_M and ternary Bonsai are further left, with retrained probes closer to bf16 than reused ones.
Layer-49 scores on CAM for each quantised model. The vertical line is bf16; filled points use the original probe, hollow points a probe retrained on the quantised model.

Below 4 bits

Q3_K_XL (~3.9 bits) is marginally smaller than NVFP4 and Q4_K_XL and shows increased degradation as expected. It loses about 0.007 AUPRC at layer 49 on both datasets. It is worth noting that this is still within the equivalence margin. A retrained probe recovers essentially all of this loss.

By around 2 bits the performance loss is more noticeable. IQ1_M's reused probe loses 0.054 AUPRC at layer 49 and falls significantly outside the margin. Retraining helps, but still leaves it about 0.02 down in deeper layers. Bonsai-27B Q1_0 is worse still: the reused probe loses 0.115 AUPRC, and retraining recovers about 85% of that. This result is expected since Bonsai-27B Q1_0 is a 1-bit model.

ModelBitsAUPRC change at layer 49, reused (CAM / Luna)Retrained (CAM / Luna)
Q3_K_XL~3.9−0.007 / −0.008−0.003 / −0.004
IQ1_M~2.0−0.054 / not run−0.022 / not run
Ternary Bonsai1.7−0.034 / −0.041−0.007 / −0.015
Bonsai-27B Q1_0~1.1−0.115 / not run−0.018 / not run

Please note that Bonsai-27B Q1_0 is built on Qwen3.6 rather than Qwen3.8, so it is compared with Qwen3.6 instead (see the appendix).

The only model to buck the trend is IQ1_M. It has more bits per weight than ternary Bonsai (2.0 against 1.7) but exhibits more performance loss. This is the case with both reused and retrained probes.

It is worth noting that quantisation performance loss was consistent across both datasets throughout.

Grid of per-layer AUROC and AUPRC changes against bf16 on CAM, one row per model from FP8 down to ternary Bonsai, with a green ±0.01 band. FP8, NVFP4 and Q4_K_XL sit on zero; Q3_K_XL dips slightly; IQ1_M and ternary Bonsai fall well below the band with the reused probe, while retrained probes sit closer to zero.
Change against bf16 at every layer on CAM, one row per model. The green band is the ±0.01 margin; filled points use the original probe, hollow points a retrained one.
Four panels of layer-49 change against bf16, CAM in blue and Luna in orange side by side for each model. Rows are AUROC and AUPRC; columns are reused and retrained probes. The two datasets track each other model by model.
Layer-49 change against bf16 on CAM (blue) and Luna (orange), for the original probe (left) and a retrained probe (right). The two datasets move together model by model.

Drift

A potential alternative measurement of probe degradation was drift. A probe is just measuring the hidden state and so if the state barely moves, the probe's output shouldn't either. However, as the results show, drift isn't reliable enough to use on its own. Cosine drift is especially poor: it reported 0.996 similarity at a layer where the probe lost AUPRC.

Scatter plots of drift against the reused probe's change for every model, layer and dataset, coloured by dataset with a marker per model, plus a zoomed column for drift below 0.2. The change grows with drift on both datasets.
Hidden-state drift against the change in the reused probe, for every model, layer and dataset. Larger drift generally means a larger loss, but at the same drift Q3_K_XL affects the probe more than NVFP4.

Why do low-bit models degrade?

A further test was done to assess if the performance loss was because low-bit quantisation destroys what the probe reads or if it just changes how/where it is stored. To test this, a linear map was fit from the ternary Bonsai's hidden states back to bf16's. The original bf16 probe was then run on these mapped states. The Luna Hallucinations dataset was used.

Layerbf16ReusedPer-dimension rescaleLinear map (eval R²)Linear map recoversCosine raw → standardised
00.2730.2500.2500.268 (0.94)75%0.996 → 0.813
70.3050.2890.2900.305 (0.81)99%0.985 → 0.849
210.3580.3370.3380.357 (0.80)97%0.953 → 0.832
350.4090.3710.3730.400 (0.77)75%0.901 → 0.780
490.4510.4100.4140.435 (0.78)61%0.845 → 0.768
630.4170.3720.3740.396 (0.76)51%0.768 → 0.741

At layers 7 and 21, the map recovers 97–99% of the lost AUPRC, about as much as retraining a probe. By contrast, rescaling each dimension separately recovers at most 10%. From this, we can conclude that the quantisation is mixing dimensions together, and so a simple rescale can't undo it. Deeper in the model the map recovers less (75% at layer 35, falling to 51% at layer 63), so at least some of the information there is truly lost.

AUROC and AUPRC by layer on CAM for bf16, ternary Bonsai with the bf16 probe reused, and ternary Bonsai with a retrained probe. The reused line sits below bf16 at every layer; the retrained line sits close to bf16.
For comparison with retraining: ternary Bonsai by layer, with the original probe reused and with a retrained probe. This is on CAM rather than Luna. Retraining recovers most of the loss.

Conclusions

  1. Down to Q4_K_XL and NVFP4 (~4.5 bits nominal), quantisation has no measurable effect on probe performance at any layer, for reused or retrained probes, on two independent datasets. Every effect is at least 2.5× inside the margin.
  2. Q3_K_XL (~3.9 bits) is the first format to move a reused probe (L49 AUPRC −0.007 on CAM, −0.008 on Luna). Retraining removes the effect.
  3. At ~2 bits and below, reused probes degrade significantly and retraining recovers only part of the loss.
  4. The damage depends on the kind of distortion, not solely on bits or drift.

Limitations

  • Only one surrogate family (Qwen) was used.
  • There was a very low positive rate for both datasets. As such token rows were sampled with enriched positives.

Next steps

NVFP4 in production. NVFP4 matched bf16 on both datasets while running about 1.5× faster, so the next step is to test it in production. This could raise surrogate throughput and cut serving cost without retraining any probes.

Combining this with other interpretability techniques. We'd like to combine this activation measurement scheme with other interpretability techniques, such as the J-space. This may be particularly useful for improving model selection in Gallery.