What RLHF Does and Does Not Do

Why a plain language model is not enough

A language model trained only to predict the next word has a different goal from the one people want. The OpenAI team behind InstructGPT put it this way in 2022: making models bigger does not inherently make them better at following a user’s intent, and large models can be untruthful, toxic or simply unhelpful. Predicting the next token on a web page is not the same objective as following instructions helpfully and safely, so the paper calls the plain objective misaligned.

Reinforcement learning from human feedback (RLHF) is the technique the paper used to close part of that gap. It has a shorter history than the chatbots built on it.

Where it came from

In 2017 Paul Christiano and colleagues at OpenAI and DeepMind showed that a system could learn a goal from human preferences between pairs of short clips of behaviour, with no reward function written by hand. They trained agents on Atari games and simulated robot movement while giving feedback on less than 1% of the agent’s interactions, and reported training new behaviours with about an hour of human time. The point was cost: a person only has to say which of two clips looks better.

The three steps in InstructGPT

The 2022 paper applied the idea to GPT-3 in three steps.

Demonstrations. A team of 40 contractors, hired after a screening test, wrote examples of the desired output for prompts submitted to the OpenAI API and for prompts they wrote themselves. GPT-3 was fine-tuned on them with ordinary supervised learning.
Rankings. The labelers compared and ranked several outputs from the model. A separate reward model was trained to predict which output they would prefer.
Reinforcement learning. The model was fine-tuned with the PPO algorithm to produce outputs that the reward model scores highly.

What it changed

The paper reports its results on its own prompt set, judged mostly by its own labelers, so read the numbers as that.

Preference. Outputs from the 1.3 billion parameter InstructGPT model were preferred to those of the 175 billion parameter GPT-3, which has over 100 times more parameters. The 175 billion parameter InstructGPT was preferred to GPT-3 85% of the time (plus or minus 3) and to GPT-3 with a few-shot prompt 71% of the time (plus or minus 4).
Truthfulness. On the TruthfulQA benchmark, InstructGPT gave truthful and informative answers about twice as often. On summarising and other closed-domain tasks it invented information not in the input 21% of the time, against 41% for GPT-3.
Toxicity. About 25% fewer toxic outputs when the model was asked to be respectful. On two bias benchmarks, Winogender and CrowS-Pairs, there was no significant improvement.

The alignment tax

Fine-tuning on human preferences made the model worse at some public benchmarks, among them SQuAD, DROP, HellaSwag and French-to-English translation. The authors call this an alignment tax. They reduced it by mixing the original pre-training objective back in during the reinforcement step, without lowering the labelers’ preference scores.

What it does not do

The paper is direct about its limits.

Whose preferences? The procedure aligns the model to the stated preferences of a specific group, mostly the labelers and the researchers, and not to any broader notion of human values.
Mistakes remain. InstructGPT can still fail to follow instructions, make up facts, give long hedging answers to simple questions and miss false premises in an instruction.
Disagreement is untested. A preliminary test with labelers who produced no training data gave similar preferences. The authors say more work is needed on broader groups and on inputs where people disagree about the right behaviour.

This article describes the 2022 recipe. Later systems have changed details, which are not covered here.

Our take: RLHF teaches a model what people prefer, which is not the same as what is true. Where the people ranking the answers cannot tell a good one from a plausible one, the signal is weak. Ask who the raters were, what they were asked and how the system was tested on things they would get wrong.

Questions and answers

What does RLHF stand for?

Reinforcement learning from human feedback. People compare a model's outputs, a reward model learns their preferences, and the model is then trained to produce outputs the reward model rates highly.

Is RLHF how a language model is trained from scratch?

No. In InstructGPT it came after pre-training: first supervised fine-tuning on human-written examples, then a reward model trained on rankings, then reinforcement learning.

Does RLHF make a model truthful?

It helped in the paper: truthful and informative answers about twice as often on TruthfulQA, and made-up content in closed-domain tasks fell from 41% to 21%. The authors also say the model still makes up facts.

Whose values does it follow?

In InstructGPT, the preferences of a specific group, mostly the 40 contracted labelers and the researchers. The paper says this is not a broader notion of human values.

Do you agree?

Take it to Chevlume: write your view, add your sources and let others answer. The closed alpha is by invitation.

Discuss this on Chevlume

No invite yet? Ask for a place

Keep reading

AI / EXPLAINER

Sources

Training language models to follow instructions with human feedback (Ouyang et al., 2022) · arXiv (OpenAI) · Accessed 4 November 2026
Deep Reinforcement Learning from Human Preferences (Christiano et al., 2017) · arXiv (OpenAI, DeepMind) · Accessed 4 November 2026

Published 4 November 2026

Discuss on X