Quantisation makes an AI model smaller by storing its weights with fewer bits, often 8-bit or 4-bit instead of 16-bit. It reduces memory use and can speed up inference, usually with some quality trade-off that must be tested on your task.
Quantisation makes an AI model smaller by storing its numbers with less precision. A model is mostly a large collection of weights. In many modern language models, those weights are stored in 16-bit precision. Quantisation stores them in fewer bits, often 8-bit or 4-bit.
Less precision means less memory. A 4-bit version of a model can be much smaller than its 16-bit version. That can make the difference between a model that needs server hardware and a model that fits on a local GPU or even a strong laptop.
The trade-off is that the model is no longer exactly the same. Rounding the weights introduces small errors. The question is whether those errors matter for your task.
Lower memory use
Weights stored in 4-bit form take roughly a quarter of the space of 16-bit weights. Google gave a concrete example for Gemma 3 QAT: the 27B model drops from 54 GB to 14.1 GB in its 4-bit quantisation-aware-trained form.
Faster inference
Inference often waits on memory bandwidth: moving model weights from memory to compute units. Smaller weights mean less data to move. On hardware that supports low-precision operations well, this can improve speed.
Local deployment
Quantisation is one reason local LLM tools such as llama.cpp, Ollama, LM Studio, and Foundry Local are practical. They usually run pre-quantised models that would otherwise be too large.
Lower cost
If a model fits on a smaller GPU, or more users fit on the same server, the cost per request can drop.
Post-training quantisation happens after the model is already trained. The weights are converted to a lower-precision format without retraining the full model. Popular approaches include GPTQ, AWQ, bitsandbytes, GGUF quantisation, and other serving-specific formats.
This is the most common route for open models because it is quick to apply and many ready-made quantised variants already exist on Hugging Face.
Quantisation-aware training, or QAT, accounts for low precision during training or additional training. The model learns while seeing the effects of quantisation, so it can keep more quality at low bit widths. QAT is more work, but it can perform better than a simple after-the-fact conversion, especially at 4 bits.
These techniques are often mentioned together, but they do different things.
Quantisation keeps the same model architecture and stores its weights more compactly. It is compression with a quality trade-off.
Model distillation trains a separate smaller student model to imitate a larger teacher model. The result is a different model.
Choosing a smaller model means starting with a model that has fewer parameters. A 7B model may be enough for classification, extraction, or local assistance even before quantisation.
In practice, teams often combine them: choose the smallest model that passes the evals, then quantise it to fit the target hardware.
Hugging Face
Model pages often list GPTQ, AWQ, bitsandbytes, GGUF, Q4, Q5, Q8, or similar variants. Those names tell you the model has been prepared for lower-precision loading or inference.
Local LLM runtimes
llama.cpp and tools built around it commonly use GGUF files. Ollama and LM Studio hide much of that complexity but still depend on quantised model files under the hood.
On-device AI
Foundry Local and similar runtimes make local inference practical by shipping or using models that fit ordinary hardware better than full-precision versions.
Test quality on your own task
A quantised model can look fine in a demo and fail on your invoice fields, Dutch support tickets, or legal clauses. Compare it with the original on a held-out test set.
4-bit is a bigger step than 8-bit
8-bit quantisation is often barely noticeable for many tasks. 4-bit can still work well, but the quality trade-off becomes more task-dependent.
Hardware matters
Lower precision does not automatically mean faster on every machine. Speed depends on GPU, CPU, memory bandwidth, runtime, and kernel support.
Context still uses memory
The model weights are not the only memory cost. Long prompts and long context windows also require runtime memory.
Official or well-tested variants are safer
A random quantised upload may not preserve quality. Prefer variants from the model maker or widely used community methods with clear eval results.
Deep learning is machine learning with neural networks that have many layers. Those layers learn increasingly abstract patterns in images, s...
Read definitionA GPU is a powerful chip built for fast, parallel calculations. It runs thousands of small operations at the same time, which makes it ideal...
Read definitionInference is the moment a trained AI model is used: new input goes in, a prediction, classification, image, or answer comes out. Training is...
Read definitionMachine learning is software that learns patterns from examples instead of following hand-written rules. You train a model on historical dat...
Read definitionModel distillation trains a smaller student model to imitate a larger teacher model. The student learns from the teacher's outputs, so it ca...
Read definition