Model distillation

What is model distillation?

Model distillation is a technique where a smaller model learns to imitate a larger one. The larger model is the teacher. The smaller model is the student. The teacher answers many example questions, and the student is trained on those answers until it behaves similarly on the target task.

The point is not to make the student know everything the teacher knows. The point is to transfer useful behaviour for a specific job. A small model that classifies support tickets, extracts fields from invoices, or writes short product descriptions does not need all the broad ability of a frontier model. It needs to be good enough at that one workflow and cheap enough to run often.

The original knowledge-distillation idea became widely known through the 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. In that setup, the student learns from the teacher's final answer and from the teacher's softer probability distribution: which wrong answers were almost plausible and which were clearly unlikely. That gives a richer learning signal than a simple right-or-wrong label.

How distillation works in practice

For business teams, distillation usually looks like a four-step loop.

1. Collect representative prompts. Use real examples from the intended task: the emails to classify, invoices to parse, product tickets to triage, or questions the assistant must answer.

2. Generate teacher outputs. Run those prompts through a stronger model. Review or filter the answers so the student is not trained on weak examples.

3. Train the student. The smaller model is fine-tuned on the teacher's outputs. The training data is often a form of synthetic data because it was generated by another model.

4. Evaluate against the teacher and the task. Test both models on held-out examples. Measure quality, latency, cost, failure modes, and how often a human still needs to intervene.

Some AI platforms now package parts of this workflow. OpenAI has described distillation flows built around stored completions, evals, and fine-tuning. Google describes Gemini distillation as a managed teacher-student service for training smaller models from stronger ones. The platform changes, but the idea stays the same: pay the large model to create a high-quality training signal, then run the small model in production.

Examples of distilled or distillation-heavy models

Gemini Flash
Google described Gemini 1.5 Flash as being trained from Gemini 1.5 Pro through distillation, transferring important capabilities into a smaller and faster model.

Phi models
Microsoft's Phi work shows the broader family of ideas around small language models, synthetic data, and teacher-generated material. The Phi-4 technical report is careful: it says earlier Phi models relied strongly on distillation from GPT-4, while Phi-4 goes beyond straightforward distillation through its data mix and post-training recipe.

DeepSeek R1 distills
DeepSeek released smaller R1-distilled models based on Qwen and Llama. Those models were fine-tuned from examples generated by the larger R1 reasoning model, giving smaller open models some of the behaviour of the larger teacher.

These examples are useful, but do not treat model names as the principle. Distillation is a pattern. The right student is the smallest model that passes your own evaluation, not the model with the most impressive release note.

Distillation versus fine-tuning versus quantisation

Fine-tuning continues training an existing model on examples so it behaves better for a task, tone, format, or domain. Distillation often uses fine-tuning as the training mechanism, but the training examples come from a teacher model.

Quantisation makes a model smaller or faster by storing its weights with fewer bits. It normally changes how the same model is represented. It does not create a new student that learns from a teacher.

Distillation creates or adapts a separate student model. The student may then be quantised as a second step. A common path is: distil to a smaller model, fine-tune on a small set of human-checked examples, then quantise for deployment.

When distillation is worth considering

High-volume repetitive tasks
If a task runs thousands or millions of times, shaving cost and latency matters. A small student can pay for the training work quickly.

Narrow tasks with clear evaluation
Distillation works best when success is measurable: did the invoice fields match, was the right category chosen, did the answer follow the schema?

Low-latency products
A chatbot, autocomplete feature, or voice workflow may need answers faster than a large model can reliably provide.

Controlled deployment
A smaller open model can sometimes run in your own environment, which matters for cost, data locality, or resilience.

What to watch out for

The teacher's mistakes travel
If the teacher hallucinates, shows bias, or uses a bad format, the student learns that behaviour. Review samples before training and include adversarial examples in evaluation.

The student is narrower than it feels
A distilled model can look strong inside the training distribution and fail quickly outside it. Test near-misses, odd inputs, missing fields, and questions just outside the intended task.

Evaluation is the control loop
Do not trust a lower bill as proof of success. Keep a held-out test set and compare the student with the teacher, a baseline prompt, and human-labelled examples where possible.

Licences and terms matter
Provider terms may restrict using model output to train competing models. Some open models allow distillation and derivatives; others come with extra conditions inherited from the base model. Check the current terms before building a student model from another model's output.

Personal data can move into weights
If teacher outputs contain personal or confidential data, the student may memorise some of it. Treat the training set like sensitive data, minimise what goes in, and keep deletion and governance requirements in mind.