Line-engraved automaton consulting an indexed archive, representing retrieval-augmented generation

RAG in Plain English: How Retrieval Actually Makes an Agent Useful

July 21, 2026

Executive Summary

Retrieval-augmented generation is not a model feature. It is a search system bolted to a writer, and the search half is where nearly all the quality lives.
The failure mode almost nobody explains: chopping documents into chunks strips each piece of the context that made it findable. A passage reading "revenue grew by 3% over the previous quarter" no longer knows which company or which quarter it belongs to.
Anthropic measured the fix. Attaching context to each chunk before indexing cut retrieval failures by 49%, and adding a reranking pass cut them by 67%.
If your entire knowledge base fits in about 500 pages, skip RAG. Paste the whole thing into the prompt and move on with your life.
RAG reduces hallucination. It does not abolish it, which is why citations and a system that can say "I don't know" matter more than your choice of vector database.

Every few months someone announces that RAG is dead, killed off by context windows large enough to swallow a novel. Then the same people ship an agent that confidently invents a refund policy, and retrieval quietly returns to the roadmap.

The useful version of this conversation is not whether retrieval survives. It is what retrieval actually does, why competent teams still get it wrong, and how you tell a working implementation from an expensive demo. Here is that, in plain English, with the jargon translated rather than deployed.

Sealed engraved brass sphere with frozen constellation patterns and locked internal gearwork, representing a language model's knowledge frozen at its training cutoff

Why a Raw Model Isn't Enough

A language model only knows what it absorbed during training, and it has no idea which parts it got wrong.

The knowledge is baked into the weights, frozen at a cutoff date, and stored as statistical association rather than retrievable fact. Ask about your refund policy and the model does not look anything up. It produces the most plausible-sounding refund policy, which is a very different operation and occasionally a legally interesting one.

You have two ways to fix this. Retrain the model on your documents, or hand it the right documents at the moment you ask the question. The industry has quietly settled this argument. In Menlo Ventures' survey of 600 enterprise IT decision-makers, 51% of production AI deployments used RAG while only 9% relied primarily on fine-tuning. Their 2025 follow-up found the same shape a year later: prompt design leads, retrieval follows, fine-tuning stays specialist.

The reason is unglamorous. Fine-tuning teaches a model how to behave. Retrieval tells it what is true today. When your pricing changes on Tuesday, you want to replace a document, not schedule a training run.

Douwe Kiela, who co-authored the original 2020 RAG paper and now runs Contextual AI, put the false choice to rest on O'Reilly's Generative AI in the Real World podcast: "Fine-tuning versus RAG is another false dichotomy. The answer has always been both."

Three-stage engraved mechanism dividing a document into tiles, sorting them through a faceted lens, and feeding the selection into a vessel, representing the prepare, retrieve and generate stages of RAG

How Retrieval-Augmented Generation Actually Works

Retrieval-augmented generation is three jobs in a trench coat: prepare the documents, find the relevant ones, then answer using only those.

Preparation. Your documents get extracted into text, split into chunks of a few hundred words, and converted into embeddings. An embedding is a list of numbers encoding meaning, arranged so passages about similar things land near each other. Those numbers go into an index, usually a vector database.

Retrieval. A question arrives and gets converted into numbers the same way. The system finds the nearest neighbors, which is a mathematical way of saying "the passages that seem to be about the same thing." Good systems run a keyword search alongside this. Semantic search understands that "reimbursement" and "refund" are cousins, but it will fumble an exact string like error code TS-999, which old-fashioned keyword matching nails every time. Running both and merging the results is called hybrid search, and it is close to free.

Generation. The winning passages get pasted into the prompt above your question, and the model answers from them.

That is it. No magic. RAG feels mysterious because all three steps are individually boring and collectively fragile, and the failure of any one of them looks identical from the outside: a confident wrong answer.

Worth saying plainly, because vendors will not: if your knowledge base is under 200,000 tokens, roughly 500 pages, you do not need any of this. Anthropic's own guidance is to put the entire corpus in the prompt and use caching. RAG earns its complexity at scale, not at pilot size.

Grid of engraved gold tiles with one tile detached and faded in empty space, representing a document chunk severed from its context and made unretrievable

Where Retrieval-Augmented Generation Quietly Breaks

The dominant failure is not the model hallucinating. It is retrieval handing the model the wrong pages, and the model doing its best with them.

Here is the mechanism most explainers skip. When you chop a document into chunks, each chunk loses the context that surrounded it. Anthropic's engineering team illustrated this with a chunk from an SEC filing reading: "The company's revenue grew by 3% over the previous quarter."

Ask "what was ACME Corp's revenue growth in Q2 2023" and that chunk should be a perfect hit. It is not, because the isolated text never mentions ACME, never mentions Q2, and never mentions 2023. The right answer is sitting in your index, correctly stored, and functionally invisible.

Multiply that across a corpus and you get a system that is right most of the time and unpredictably wrong the rest. Which is worse than useless, because people stop checking.

The fix is to reattach the context before indexing: prepend a sentence to each chunk explaining where it came from. Anthropic measured what that buys you across codebases, fiction, and scientific papers, using the rate at which the correct passage fails to appear in the top 20 results:

Contextual embeddings alone: 35% fewer retrieval failures (5.7% down to 3.7%)
Adding contextual keyword search: 49% fewer (down to 2.9%)
Adding a reranking pass: 67% fewer (down to 1.9%)

The cost of generating that context, with prompt caching, was $1.02 per million document tokens. The single highest-leverage improvement available to most RAG systems costs roughly a sandwich.

Upstream of all of it sits a problem nobody wants to own. If your PDF extractor mangles the tables and drops the headings, nothing downstream can recover. Kiela is blunt about the ordering: "If your information extraction is bad, you can chunk all you want and it won't do anything. Then you can embed all you want, but that won't do anything." He also named where projects go to die: "You need to build for production, not for the demo. You need to make it work on a million PDFs. We see a lot of projects die on the way to productization."

I have made this argument before in a different register when writing about turning company documents into something an agent can actually use. The document layer is the product. The model is a commodity you rent.

Brass balance scale weighing tethered source tablets against a deliberately empty pan, representing grounded citations and a system able to say it does not know

Grounding and Citations That Build Trust

Retrieval reduces hallucination. It does not eliminate it, and anyone promising otherwise is selling something.

A general-purpose model is trained to be helpful, fluent, and willing to fill gaps. Hand it five correct passages and a question they only partly answer, and it will bridge the gap with something reasonable-sounding. The context was right. The behavior was wrong.

Better systems change what the model is allowed to do, not just what it sees. Kiela's team trains models to stay inside the retrieved context and, crucially, to decline: "The language models are very good at saying, 'I don't know.' That's really important. Our model cannot talk about anything it doesn't have context on."

That gives you two evaluation tests worth more than any benchmark. Ask it something your documents genuinely do not cover, and see whether it admits it or improvises. Then check whether every claim carries a citation you can click. Citations are not decoration. They are the only mechanism that lets a human catch the system being wrong, and a system without them is asking for trust it has not earned.

What a Solid Setup Looks Like

A good implementation is unremarkable in structure and obsessive about evaluation.

Extract documents properly, including tables and hierarchy, before anything else. Chunk with context attached rather than slicing blindly at a token count. Run hybrid retrieval, semantic and keyword together. Cast a wide net, then rerank aggressively to a small, high-quality set. Ground the generator so it answers from the retrieved text or says it cannot. Show citations. Then build an evaluation set of real questions from real users and measure whether the right passage actually surfaces, because every choice above is a knob, and knobs turned by intuition tend to get turned wrong.

Notice what is absent: the brand of vector database. It matters far less than the order of operations, which is roughly the same lesson as reliability being an engineering property rather than a model property, and the same reason bad data defeats good tooling.

As for the death notices, Kiela offered the tidiest rebuttal to the claim that agents made retrieval obsolete: "I would still call that RAG. The agent is the generator. You're augmenting the G with the R. If you want to get these systems to work on top of your data, you need retrieval." And on long context, an observation with more teeth than most: "It would be nice if longer context actually worked. You will still need RAG."

Retrieval did not die. It moved down a layer and stopped being a buzzword, which is what useful infrastructure does.

Wide engraved procession of archive drawers receding toward a single luminous diamond, representing retrieval narrowing a large corpus to one relevant answer

Frequently Asked Questions

What Is Retrieval-Augmented Generation?

Retrieval-augmented generation is a technique where a language model retrieves relevant passages from an external knowledge base and uses them as context when generating an answer. Instead of relying only on what it absorbed during training, the model works from documents you control and can update. The term comes from a 2020 paper by Lewis and colleagues, which combined a model's internal memory with an external, swappable index.

How Does RAG Reduce Hallucinations?

It grounds answers in retrieved source text rather than statistical recall, and it lets the system cite where each claim came from. It reduces hallucination substantially but does not remove it, because a model given correct context can still over-interpret or fill gaps. Systems trained to answer only from context, and to say "I don't know" otherwise, close much of the remaining distance.

What Is the Difference Between RAG and Fine-Tuning?

Fine-tuning adjusts the model's weights to change how it behaves: tone, format, or task-specific skill. RAG leaves the model untouched and changes what information it sees at question time, so updating knowledge means replacing a document rather than running another training job. Menlo Ventures found 51% of production deployments using RAG against 9% relying mainly on fine-tuning, though the two combine well.

What Is a Vector Database, and Do I Need One?

A vector database stores text as numeric embeddings and retrieves passages by meaning rather than exact wording. You need one when your corpus is too large to fit in a prompt. Below roughly 200,000 tokens, about 500 pages, you can skip retrieval infrastructure entirely and include everything in the prompt.

Do Long Context Windows Make RAG Obsolete?

No. They solve an overlapping problem, but sending an entire corpus on every query is slow, expensive, and degrades as context grows. Most production systems use retrieval to narrow the field, then hand a long-context model the finalists.

What Makes a RAG Implementation Actually Good?

Clean document extraction, chunks that retain surrounding context, hybrid semantic plus keyword retrieval, a reranking pass, a generator constrained to the retrieved material, visible citations, and an evaluation set of real user questions. Anthropic's measurements suggest contextualized chunks plus reranking alone cut retrieval failures by 67%, a larger gain than most teams get from changing models.

If you want a straight answer about whether retrieval is the right tool for what you are building, or whether you are about to over-engineer a problem that fits in a prompt, that is exactly what our System Review Diagnostic is for.

References

Back to Blog