Post-training

What is post-training?

Post-training is everything a lab does to a model after it has finished reading its training text. Pretraining leaves you with something that predicts the next word very well and nothing else. Ask it a question and it will continue your question instead of answering it, and it has no sense of when to stop or when to refuse. Post-training turns that text predictor into an assistant.

Think of hiring someone. Pretraining is the years of reading and schooling they did before you met them. Post-training is the induction week: this is how we answer a customer, this is the format we use for a quote. The knowledge was already in their head. What changes is what they do with it.

The word covers a pipeline, not one technique, and it is where much of the visible difference between two model versions now shows up.

The stages, in order

Almost every published recipe makes the same three moves, and each one changes something different.

1. Supervised fine-tuning on demonstrations. People write good answers to a pile of prompts and the model is trained to copy them. This teaches the shape of the job: answer the question, use this structure, stop when you are done. The Tülu 3 recipe Ai2 published in November 2024 is one of the few fully documented examples in public, built on a pool of roughly 940,000 curated prompts.

2. Preference learning from comparisons. The model writes several answers to the same prompt and a rater says which is better. Those comparisons shape tone, helpfulness and refusals, the things nobody can write a rule for. This is the RLHF step, and it is where a model's manners get set.

3. Reinforcement learning against a checker. The model answers, a program decides whether the answer is right, and the model is trained on that verdict. No human sits in this loop, which is why it runs at a scale the previous stage never reaches.

Labs do not run the same number of stages or use the same names. Qwen3, documented by Alibaba in May 2025, lists four post-training stages and finishes with a general reinforcement learning stage covering more than twenty task types. Read any single lab's diagram as one worked example, not as the industry standard.

Reinforcement learning with verifiable rewards

This is the newest of the three stages and the one that has moved fastest. Ai2, which named it in November 2024, described it as replacing the reward model with a verification function. In plain terms: instead of a second model guessing what score a person would give, a program checks the answer and hands back a reward only if it passes.

What gets checked depends on the task. Does the test suite run green? Is the maths result the right number? Does the code compile? DeepSeek's R1 work, published in January 2025, ran on exactly this: the model has to put its final answer in a box so that a rule can read it. They wrote that they avoided a learned reward model on purpose, because such a model "may suffer from reward hacking in the large-scale reinforcement learning process".

The consequence matters more than the mechanism. This only works where correctness is machine-checkable. Maths has a right answer, code either runs or it does not, and an email to a supplier has neither. That is why coding and maths improved fastest across model versions and open-ended writing did not. Nothing about writing got harder. There is simply no program that can hand out the reward.

It is the same shape as a verification loop around an agent, run somewhere else: a verification loop closes during one work session, verifiable rewards close during training, millions of times over.

Preference learning versus an automatic checker

Both are reinforcement learning. The dimension that separates them is where the training signal comes from.

From a person. A rater compares two answers and picks one. The signal is an opinion, so it reaches anything a reader can judge: is this polite, should the model have refused. The price is that opinions are slow and expensive to collect, they differ from rater to rater, and the model can learn to please the rater rather than help them.

From a program. A checker reads the output and returns pass or fail. The signal is a fact about the answer, so it is cheap, repeatable and available in unlimited quantity. The price is that it only exists where somebody could write the rule in the first place.

Labs mix the two per task rather than picking one. Qwen3's final stage uses rule-based rewards where a rule fits, a second model scoring against a written reference answer where it does not, and a preference-trained reward model for the rest.

Mid-training, and what labs actually mean by it

Mid-training is the name some teams give to a stage between the bulk of pretraining and everything above. The model keeps training on plain text, but on a smaller and better mix: more maths, more code, longer documents. It is still learning what the world contains, not yet how to behave.

The word is not settled. Ai2 uses it for the second stage of OLMo 2, published at the end of 2024, where the model trains on higher-quality text plus synthetic data written to patch weak spots in maths. Alibaba does something recognisably similar for Qwen3 but files it inside pre-training under the name reasoning stage. So when a vendor says they improved the mid-training, ask what data they added and what it was meant to fix.

Two tasks side by side

Reconciling the VAT return against the ledger. There is a right answer and you already own the check: run the model's script against last year's closed books and see whether the totals land. That shape is what a lab can train on at scale, because the same check that satisfies your accountant hands out a reward a million times with nobody watching. It is why new model versions tend to show real gains on this kind of work.

Writing the mail that tells a customer their delivery slipped by two weeks. No program can score that mail. The best training signal available is a rater's preference, or a written rubric a second model grades against, and both carry the taste of whoever wrote them. Post-training can teach the model to follow your instructions and hold a register. It cannot practise a million attempts against "the customer was not annoyed", because nothing can tell it whether the customer was annoyed.

So where you can write the check, expect the model itself to keep getting better. Where the only judge is a human reaction, your prompt, your examples and the context you feed it are the part that makes the difference.

What to watch out for with post-training

It changes behaviour, not knowledge. A model that never saw your industry's document layout during pretraining will not learn it here. For everything it never read, you need retrieval or a longer prompt, not a better-behaved model.

Answers that satisfy the checker are not always right answers. Optimising against a program teaches the model to make that program say yes, and the shortest route there is not always the honest one. In November 2025 Anthropic published an experiment run in its own production coding environments. Once a model had learned to reward hack there, it went on to behave badly in unrelated ways: faking agreement with its training, helping people who were up to no good, sabotaging work. Safety training in chat form afterwards cleaned up the chat evaluations and left the problem standing on agentic tasks.

A new version can be worse at something the old one did. OpenAI withdrew a GPT-4o update in April 2025 after it turned flattering to the point of agreeing with bad ideas, and wrote that the update had focused too much on short-term feedback. If a model release breaks something for you, that is a normal outcome, not a misunderstanding on your side.

"Better at coding" does not mean better at your job. Improvements land where the training signal was, so a release note about coding says nothing about how the model reads your purchase orders. Keep ten or twenty of your own real cases with the answers you expect, and rerun them on every version you weigh up.

You are almost never doing any of this. The slice a customer can buy is fine-tuning, roughly the first stage of the pipeline run on your own examples, and it is worth the cost only for narrow, repetitive work. The rest is not something you run. It is the reason the models you rent behave the way they do.