AI 4 November 2026 6 MIN READ
How a Language Model Is Trained
From tokens to chat: next-token prediction, human feedback, scaling laws and why models hallucinate, with the numbers from the original papers.
THE SHORT ANSWER
A language model first learns to predict the next token across a vast amount of text. Then it is fine-tuned with human examples and rankings so it follows instructions. Most mistakes come from those two stages: facts that patterns cannot predict, and training that rewards guessing.
On this page
Step one: turn text into tokens
A language model does not read letters or words. It reads tokens: pieces of text from a fixed vocabulary. A common method, byte pair encoding, was adapted for neural networks in 2015. It builds a vocabulary of variable-length chunks, so even a rare word can be spelled out from smaller pieces. OpenAI’s help pages give a rule of thumb for English: about four characters, or three quarters of a word, per token, so 100 tokens are roughly 75 words. It is an estimate, and other languages differ.
Step two: predict the next token
The core of training is a simple task. Pretraining is next-token prediction on huge amounts of text. OpenAI describes it plainly: there are no true or false labels, and the model only sees examples of fluent language. Its job is to guess what comes next, over and over, and the parameters are adjusted whenever it guesses wrong.
The machinery for this is the transformer, introduced in a 2017 paper whose title is its main idea: “Attention Is All You Need”. It dropped recurrence and convolutions and relied only on attention, which lets every token look at every other token in the text. The original model was built for machine translation, not chat. It reached 41.8 BLEU on English–French after 3.5 days on eight GPUs.
Attention has a price. The paper notes that its cost per layer grows with the square of the sequence length. That is one reason long inputs are expensive.
Step three: learn to follow instructions
A model that only predicts the next token is not yet an assistant. The InstructGPT paper from OpenAI, published in 2022, says it directly: the standard pretraining objective, predicting the next token on web pages, differs from the goal of following instructions helpfully and safely.
Its fix has three steps. First, humans write example answers and the model is fine-tuned on them. Second, humans rank several model answers and a reward model learns those preferences. Third, the model is trained with reinforcement learning against the reward model. This is reinforcement learning from human feedback, or RLHF. The idea predates language models: a 2017 paper trained agents from human choices between pairs of short clips, using feedback on under 1% of the agent’s interactions.
The human data was surprisingly small. InstructGPT used a team of about 40 contractors, and the three datasets held roughly 13,000, 33,000 and 31,000 prompts. Yet human evaluators preferred outputs from the 1.3-billion-parameter InstructGPT to those from the 175-billion-parameter GPT-3, even though InstructGPT has a hundred times fewer parameters.
It has costs too. The paper reports small drops on some public benchmarks, which it calls an alignment tax, and it uses a penalty to keep the model from drifting too far from its starting point while it chases the reward.
How big, and on how much text
How much should you spend on a larger model, and how much on more text? In 2020, OpenAI researchers found that loss falls as a power law with model size, dataset size and compute, across more than seven orders of magnitude. Their advice leaned towards very large models trained on relatively modest data.
DeepMind’s Chinchilla paper corrected this in 2022. After training more than 400 models, from 70 million to over 16 billion parameters and from 5 billion to 500 billion tokens, they found that model size and training tokens should grow together: for every doubling of model size, double the tokens. Existing large models, they said, were significantly undertrained. Their test was Chinchilla itself, with 70 billion parameters trained on 1.4 trillion tokens. It used the same compute as the 280-billion-parameter Gopher, but with a model four times smaller and four times as much data.
The authors add a caveat: this is the best split for a training budget, not for the cost of running the model afterwards.
Why models make things up
Hallucination means a model confidently produces an answer that is not true. Research points to two well-understood causes rather than a mysterious glitch.
- Some facts cannot be predicted from patterns. OpenAI’s 2025 explanation gives an arbitrary fact, such as a pet’s birthday, as an example. Next-word prediction will err on facts that appear rarely and follow no pattern, while spelling and bracket mistakes fade as models grow.
- Training and testing reward guessing. A 2025 paper by Kalai and colleagues argues that models guess when unsure because the usual procedures reward a guess over admitting uncertainty. The proposed fix is to penalise confident errors more than uncertainty and give partial credit for saying “I don’t know”.
Other mitigations are documented. On closed tasks, InstructGPT made up information that was not in the input about half as often as GPT-3: 21% of the time against 41%. Retrieval-augmented generation connects the model to a searchable source, and the original RAG paper found that its models hallucinated less than a baseline. Bigger is not automatically more truthful, either: on the TruthfulQA test, the best model was truthful on 58% of questions against 94% for people, and the largest models were generally the least truthful, because they imitate popular misconceptions.
Context windows
The context window is how much text, in tokens, the model can attend to at once. GPT-3’s few-shot prompts had to fit in 2,048 tokens. In March 2024 Google reported that Gemini 1.5 retrieved information almost perfectly in tests of up to at least 10 million tokens. A big window is not the same as using it well: in the “Lost in the Middle” study, accuracy was often highest when the relevant information was at the start or end of the input, and fell significantly when it was in the middle.
What to watch next
- Data, not just size: how much text of what quality, and how it is filtered.
- Post-training: this is where assistants get their behaviour, with far less data than pretraining.
- How models are scored: benchmarks that reward guessing produce models that guess.
- Long context in practice: whether the model uses the middle of a long document as well as the ends.
Our take: a prediction engine, taught manners. A chat model is a next-token predictor trained on a large share of the written web and then shaped with comparatively little human feedback. Understanding those two stages explains most of what it does well, and most of how it fails.
Questions and answers
What is a token?
A piece of text from the model's fixed vocabulary, often part of a word. For English, OpenAI's rule of thumb is about four characters, or three quarters of a word, per token.
What is RLHF?
Reinforcement learning from human feedback. People rank model answers, a reward model learns those preferences, and the language model is then trained to score well against it. It turns a text predictor into an assistant.
Why do AI models hallucinate?
Partly because some facts cannot be predicted from patterns, and partly because training and testing reward a confident guess over admitting uncertainty. Retrieval from a trusted source and better scoring both reduce it.
Is a bigger model always better?
No. DeepMind's Chinchilla work found that many large models were undertrained, and that model size and training text should grow together. A 70-billion-parameter model trained on more data beat a model four times larger.
Do you agree?
Take it to Chevlume: write your view, add your sources and let others answer. The closed alpha is by invitation.
Keep reading
The Mechanical Turk: A Chess Machine Hoax
A chess machine from 1770 that was really a hidden person. How the trick worked, who it beat, and how its name ended up at Amazon.
AI / DISCOVERYELIZA, Deep Blue and AlphaGo
Three machines that surprised their makers: a 1966 script that only reflected, a chess engine that calculated and a Go player with a famous move 37.
AI / EXPLAINEROpen Source vs Open Weights in AI
What 'open' really means for AI models: the Open Source AI Definition, open weights, restrictive licences and why the difference changes what you may do.
Sources
- Neural Machine Translation of Rare Words with Subword Units · arXiv (Sennrich, Haddow, Birch) · Accessed
- What are tokens and how to count them? · OpenAI Help Center · Accessed
- Why language models hallucinate · OpenAI · Accessed
- Training language models to follow instructions with human feedback · arXiv (Ouyang et al., OpenAI) · Accessed
- Attention Is All You Need · arXiv (Vaswani et al., Google) · Accessed
- Deep reinforcement learning from human preferences · arXiv (Christiano et al.) · Accessed
- Scaling Laws for Neural Language Models · arXiv (Kaplan et al., OpenAI) · Accessed
- Training Compute-Optimal Large Language Models · arXiv (Hoffmann et al., DeepMind) · Accessed
- Why Language Models Hallucinate · arXiv (Kalai, Nachum, Vempala, Zhang) · Accessed
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv (Lewis et al.) · Accessed
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · arXiv (Lin, Hilton, Evans) · Accessed
- Language Models are Few-Shot Learners · arXiv (Brown et al., OpenAI) · Accessed
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context · arXiv (Gemini Team, Google) · Accessed
- Lost in the Middle: How Language Models Use Long Contexts · arXiv (Liu et al.) · Accessed
Published
Discuss on X