21 August 2026

Quantisation - a Market Paradigm Shift

Modern iterations of LLMs have clearly demonstrated their penchant for creating market value. AI has replaced 80 million jobs since 2020 (1), and we have seen start-ups begin to pivot from offering the highest-reasoning models for free trials to ruthlessly capitalising on market share via subscription models. Off-the-balance-sheet billion-dollar deals are being made specifically to fund AI hardware, which suggests that the market wants, or will soon want, token efficiency. With this change comes a shift in focus for the AI research community: how do we reduce the costs associated with using AI?

Most AI requests are painfully trivial. You may have felt the pain of seeing an eccentric typist fill their context window with "please" and "thank you". At a production level, the same thing is being done: requests are sent to OpenAI endpoints that ask for a one-word response, such as "yes" or "no", or to pick a choice out of a fixed number of options. Before, AI start-ups could afford to send every request to the highest-reasoning model. Now, their pitches will need to justify to investors how their business model can maintain token efficiency.

What Is Quantisation?

What if I told you that a 30-billion-parameter frontier model could be run on your home machine?

An overview of how quantisation truncates a floating-point mantissa and changes its binary representation. — Quantisation reduces the precision of model weights while preserving their useful structure.

Quantisation is the process of intelligently altering model weights to keep a similar output while making the act of running the model faster. A simple example goes as follows: weights are stored as floating-point numbers, which themselves are stored as mantissa and exponent. This is essentially scientific notation, such as

1.234 \times 10^5,

where the mantissa is $1.234$ and the exponent is $5$ . By truncating the mantissa, here rounding down to $1.2$ , each number in the resulting quantised model takes less effort to multiply:

1.234 \times 10^5 \longrightarrow 1.2 \times 10^5.

A comparison of traditional floating-point formats with NVIDIA's NVFP4 block floating-point format. — NVFP4 shares an exponent across a block, reducing metadata overhead and improving memory efficiency.

The latest of these developments is NVIDIA's NVFP4 format, which stores floating-point values with a standard specifically optimised for machine learning. By introducing bits that act as an exponent for a whole block of data instead of a single value, attention operations can be made significantly more efficient.

Does Quantisation Really Help?

Hugging Face allows anyone to submit their models, and for Gemma 4 alone there are over 300 user-submitted quantised versions. Not all of these serve the explicit purpose of increasing efficiency, but among the most popular are creations from NVIDIA, AMD, and Unsloth. Tests were performed on an RTX PRO 6000, which has native support for the NVIDIA quantisation format.

Bar chart comparing generation throughput for a Google baseline, NVIDIA NVFP4, AMD MXFP4, and Unsloth GGUF. — Generation throughput relative to the Google floating-point baseline.

Here we can see that generation throughput for NVIDIA NVFP4 reaches $1.64\times$ the Google baseline. But how does quality compare?

Bar charts comparing F1, precision, recall, average precision, and ROC AUC across quantised model formats. — Quantisation changes headline probe metrics only marginally across the tested formats.

It is clear that these various levels of quantisation only marginally affect model quality.

How We Use AI

Part of what we do at Telluvian is white-box hallucination detection. Given some open-weight model, how can you determine if what it is saying is truthful? If a fact itself is spuriously learned during the training process, then there is little hope to correct it, but more commonly models hallucinate new knowledge where nothing was learned in the first place. This seems inevitable when you consider that the training criterion for the model is to make its output as convincingly similar to real text as possible.

When a model generates text, the input filters through a series of hidden attention layers before finally streaming as a sequence of output tokens. This means that at runtime, these tokens depend entirely on the hidden states before them. We can utilise these hidden states to read what the model is saying before it finishes generating.

These hidden states contain signals about what the model is doing. By collecting examples of truthful and hallucinated outputs, we can train a small classifier called a probe to distinguish between their internal representations. At runtime, this gives us a hallucination score for each token as it is generated.

A six-step diagram showing how hidden states from an open-weight model are used to train a hallucination probe. — A probe learns to distinguish hallucinated from truthful responses using hidden states from an open-weight model.

However, this requires access to the model's internals, which closed models like GPT and Claude do not expose. To get around this, we use proxy white-box interpretability: replaying their outputs through an open-weight model and inspecting its hidden states instead. It stands to reason that, to achieve the same result, the open-weight model must go through a similar internal process, and to generate this nonsensical output it must force itself to hallucinate.

A diagram showing a prompt sent to a closed model and an open-weight proxy model whose hidden states can be inspected. — Proxy white-box interpretability uses an open-weight model to inspect a closed model's output indirectly.

Conclusion

In the current market, organisations are drifting away from free-trial and unlimited-token business models, instead favouring token cost, speed, and efficiency. This reflects the market trend that users are starting to demand cheaper token prices.