RLHF (Reinforcement Learning from Human Feedback)

What is RLHF?

RLHF, short for reinforcement learning from human feedback, is a way to train an AI model so its answers line up with what people actually prefer. People rank different answers the model produces, and those rankings teach the model which kind of response to aim for.

It is the training step that turned raw language models into the chat assistants and other generative AI tools most people use today. A base model that has only read text is good at predicting the next word, but it will happily produce a rude, rambling, or unsafe answer. RLHF nudges it towards responses that are helpful and reasonable to read.

The technique became widely known through InstructGPT, a set of models that OpenAI tuned this way in 2022. The same recipe sits behind the popular chat assistants that followed.

Why RLHF exists

"Write a good answer" is not something you can express as a simple maths formula. Whether a reply is clear, polite, or safe is a matter of human judgement, and there is no single correct output to train against.

Ordinary supervised training needs a labelled right answer for every example. For open questions there is no such thing. RLHF gets around this by learning from comparisons instead. A person does not have to write the perfect reply, they only have to say which of two replies is better. Collecting that kind of preference is far easier and scales to a lot of examples.

How RLHF works

RLHF usually runs in three stages, each building on the one before.

1. Supervised fine-tuning. Human writers produce good example answers to a range of prompts, and the base model is fine-tuned on them. This gives the model a first sense of the instruction-following style before any ranking happens.

2. Training a reward model. The model generates several answers to the same prompt. People rank those answers from best to worst. Those rankings train a second model, the reward model, whose only job is to look at an answer and predict the score a person would give it.

3. Reinforcement learning. The main model now practises against the reward model. It writes an answer, the reward model scores it, and the main model adjusts to earn higher scores. Over many rounds it learns to produce the kind of answer people ranked highly. The algorithm most often used for this step is called PPO, though newer methods now compete with it.

One result made the value clear. OpenAI reported that people preferred answers from a 1.3 billion parameter InstructGPT model over answers from the much larger 175 billion parameter GPT-3, purely because the smaller model had been through this process.

RLHF versus plain fine-tuning

Plain fine-tuning shows the model a set of correct answers and trains it to copy them. It works well when there is a clear target, such as classifying an email or pulling a date out of an invoice.

RLHF is for cases where "correct" is a spectrum. Instead of copying fixed answers, the model is scored on how well its output matches human preference, so it can improve on qualities like tone, helpfulness, and refusing an unsafe request. In practice the two are used together: supervised fine-tuning first, RLHF on top.

What to watch out for with RLHF

The reward model is only as good as the rankings. If the people doing the ranking are rushed, inconsistent, or not representative, the model learns their blind spots. Preference data quality sets the ceiling on everything above it.

Models learn to game the score. A model can find answers that the reward model rates highly but that people do not actually like, such as replies that sound confident while saying little. This is known as reward hacking, and it needs watching.

It can make a model overcautious. Push too hard on safety rankings and the model starts refusing harmless requests. Balancing helpfulness against caution is an ongoing tuning job, not a one-time setting.

It shapes behaviour, not knowledge. RLHF changes how a model responds, not what facts it knows. It does not fix a model that is missing information, so teams usually add separate guardrails on top to catch unsafe inputs and outputs that training alone misses.