Retrieval-Augmented Generation, Explained Plainly

The problem retrieval solves

A language model keeps what it knows inside its parameters. The 2020 paper that named retrieval-augmented generation (RAG), by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, lists the weak points of that design: such models cannot easily expand or revise their memory, cannot straightforwardly show where a prediction came from, and may produce “hallucinations”.

The authors’ answer was to pair two kinds of memory. The parametric memory is the pre-trained model itself. The non-parametric memory is an external collection of documents that the model can look things up in before it answers.

How the original system worked

The paper is specific, which makes it a good picture of the idea.

The library. A December 2018 dump of Wikipedia, cut into disjoint chunks of 100 words, giving about 21 million passages. Each passage was turned into a vector, a list of numbers, by a document encoder based on BERT, and all of them went into one searchable index.
The retriever. Given a question, a second encoder turned it into a vector too, and a maximum inner product search returned the top-K closest passages.
The generator. BART-large, a sequence-to-sequence model with 400 million parameters. The question and a retrieved passage were simply joined together as its input.
Training. Retriever and generator were trained together, with no instruction about which passage to retrieve. The document encoder and the index were kept fixed; only the query encoder and the generator were tuned.

The paper compares two variants. RAG-Sequence uses the same passage for the whole answer. RAG-Token can draw on a different passage for each word it writes.

What it bought

On open-domain question answering, RAG set a new state of the art on Natural Questions, WebQuestions and CuratedTrec. On TriviaQA it did so only on the split that allows comparison with T5. On Natural Questions, RAG-Sequence scored 44.5 exact match, against 34.5 for the closed-book T5-11B, which has no retrieval.

For writing rather than answering, the authors asked human judges to compare 452 pairs of Jeopardy-style questions generated by BART and by RAG. BART was judged more factual in 7.1% of cases and RAG in 42.7%. Retrieval also helped when it missed: on Natural Questions, RAG produced a correct answer in 11.8% of the cases where no retrieved passage contained it, which an extractive system could not do.

The update trick

The most practical result is about freshness. The authors built a second index from a December 2016 Wikipedia dump and asked 82 questions of the form “Who is the President of Peru?” about world leaders who had changed between the two dates.

With the matching index, RAG answered 70% correctly for the 2016 leaders and 68% for the 2018 leaders.
With the wrong index, accuracy fell to 12% (2018 index, 2016 leaders) and 4% (2016 index, 2018 leaders).

So knowledge can be updated by replacing the library, with no further training. The reverse is also true: a stale library produces stale answers, and the model has no way to notice.

Where it goes wrong

Retrieval is not automatically better than simpler search. In the paper’s own ablations, a plain word-overlap retriever (BM25) did best on FEVER, a fact-checking task whose claims are heavy on names. Learned retrieval won on the other tasks, and mattered most for open-domain question answering.

Finding a passage is also not the same as using it. A 2023 study from Stanford, Berkeley and Samaya AI, “Lost in the Middle”, tested models on multi-document question answering and key-value retrieval. Performance was often highest when the relevant information sat at the beginning or the end of the input, and dropped significantly when it sat in the middle, even for models built for long inputs. The models tested were those of 2023, so check the position effect on any model you actually use.

What to ask of a RAG system

What is in the library, and as of when? The 2016 and 2018 results show how much the date matters.
Can you see the passages? The paper calls the memory human-readable and human-writable, because it is made of raw text that people can inspect and edit. A system that hides its sources loses that advantage.
How were the passages chosen? A better retriever, or a simple one, can matter more than a bigger generator.
Where do the passages go in the prompt? Position can change the result.

Our take: RAG moves the problem from the model to the library. An answer can be no better than what was indexed and what was found, so the useful questions are about the index, the search and the date, not only about the model.

Questions and answers

What does RAG stand for?

Retrieval-augmented generation: a language model is given passages retrieved from an external collection of documents and writes its answer using them. The name comes from a 2020 paper by Lewis and colleagues.

Is RAG the same as fine-tuning?

No. Fine-tuning changes a model's parameters. In the original RAG work, the model's knowledge could be updated by replacing the document index, with no further training.

Does retrieval stop hallucinations?

It reduced them in that study: judges found RAG more factual than a BART baseline in 42.7% of pairs against 7.1%. It did not remove them, and a stale or wrong index gives wrong answers.

Does it matter where the retrieved text goes in the prompt?

It can. A 2023 study found that models often used information at the start or end of a long input better than information in the middle.

Do you agree?

Take it to Chevlume: write your view, add your sources and let others answer. The closed alpha is by invitation.

Discuss this on Chevlume

No invite yet? Ask for a place

Keep reading

AI / EXPLAINER

Sources

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) · arXiv (Facebook AI Research, UCL, NYU) · Accessed 4 November 2026
Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) · arXiv (Stanford, UC Berkeley, Samaya AI) · Accessed 4 November 2026

Published 4 November 2026

Discuss on X