In short
Retrieval-augmented generation (RAG) is a pattern where a system searches your own documents for relevant passages and supplies them to a language model as context, so the answer is grounded in your material rather than in the model's training data. It is the correct tool when answers must reflect current, private, or citable information. Almost every RAG failure is a retrieval failure: the model answered well from the wrong passages.
What RAG is
Retrieval-augmented generation is a pattern where a system searches your own content for relevant passages, hands them to a language model as context, and asks it to answer using that material.
The model does not memorise your documents. It receives the relevant portion at question time and answers from it — which is why the same system reflects a document updated five minutes ago.
Why not just fine-tune?
This is the most common early mistake, so it is worth being blunt.
Fine-tuning changes how a model behaves. Retrieval changes what it knows. If you want consistent structure, tone, or task behaviour, fine-tune. If you want current, accurate, citable facts, retrieve.
Fine-tuning for facts fails on every practical dimension: you retrain whenever content changes, you cannot cite a source, you cannot revoke a document, and the model still blends training data with your material in ways you cannot inspect. Retrieval sidesteps all four.
How it actually works
Four stages, and the interesting problems are all in the first two.
1. Chunking. Documents are split into passages. This unglamorous step determines more about quality than anything downstream. Chunks that are too small lose the context that makes them meaningful; too large and they dilute the signal and waste context window. Splitting mid-table or mid-clause produces passages that are actively misleading.
2. Indexing. Each chunk is converted to an embedding — a numerical representation placing similar meanings near each other — and stored. This is what lets a question match a passage that shares no keywords with it.
3. Retrieval. The question is embedded and the closest chunks are returned. In production this is usually hybrid: semantic similarity combined with keyword matching, because pure semantic search is weak on names, codes, and exact terms.
4. Generation. Retrieved passages go to the model with the question and an instruction to answer from the supplied material and say when it cannot.
Where RAG systems actually fail
Almost every failure is a retrieval failure. The model answered perfectly well from the wrong passages. Teams debug the prompt for a week before checking what was retrieved. Log the retrieved chunks for every query — the answer to "why did it say that?" is nearly always visible there.
The absent-evidence problem. Retrieval returns what matches the question, not what contradicts it. Ask "is this approach approved?" and it surfaces passages about the approach — not the memo revoking approval, which may not resemble the question at all. Absence of a contradiction in retrieved context is not evidence of agreement.
Stale and superseded content. If your corpus contains three versions of a policy, retrieval may return the oldest. Recency and version awareness have to be built deliberately; similarity has no opinion about which document is current.
Confident synthesis across chunks. Given two passages, a model will happily produce a coherent answer that neither supports — combining a figure from one with a qualification from the other. This is hallucination wearing citations, and it is the hardest failure to spot because the sources are real.
Permission leakage. If the index contains documents a user should not see, retrieval will happily surface them. Access control belongs at retrieval time, filtered per user — not as an instruction to the model.
Building one that stays accurate
Evaluate retrieval separately from generation. Build a set of real questions with the passages that should be retrieved, and measure whether they are. If retrieval is wrong, no amount of prompt work helps.
Show sources in the interface, and make them specific. Link to the passage, not the document. A citation the user cannot check is decoration — and one that does not support the claim beside it is worse than none.
Instruct the model to refuse. Explicitly: answer only from the supplied material, and say plainly when it does not contain the answer. Then test that it actually does, because a helpful model will fill gaps.
Clean the corpus. Duplicates, drafts, and superseded versions cause more bad answers than any model choice. This is unglamorous and it is most of the work.
Filter by permission at query time, before retrieval, per user.
Re-evaluate when anything changes. Model updates change behaviour underneath you. A fixed evaluation set is how you find out before your users do.
When RAG is the wrong answer
If your content fits comfortably in a modern context window and does not change often, just include it — you avoid an entire retrieval system and its failure modes. If your questions need computation, aggregation, or joins across records, that is a database query, not retrieval. And if the corpus is genuinely small, plain search with good filters may serve users better than a generated answer.
RAG is a good default for grounded question-answering over a body of documents. It is not a default for everything.
If you are weighing a RAG build and want an honest read on whether retrieval is the right shape, book a call.
Common questions
What is retrieval-augmented generation in simple terms?
A pattern where the system searches your own documents for passages relevant to a question, gives those passages to a language model as context, and asks it to answer from them. The model is not trained on your documents — it receives the relevant part at question time, so updates take effect immediately.
Is RAG better than fine-tuning?
They solve different problems. Fine-tuning changes how a model behaves — its format, tone, or task handling. Retrieval changes what it knows. For current, private, or citable facts, retrieval is correct; fine-tuning for facts means retraining on every content change, with no ability to cite or revoke a source.
Why does my RAG system give wrong answers?
Almost always because retrieval returned the wrong passages — the model then answered them perfectly well. Log the retrieved chunks for every query before touching the prompt. Common causes are poor chunking, stale or duplicated documents in the corpus, and pure semantic search failing on names and exact terms.
What are the security risks of RAG?
The main one is permission leakage: if the index contains documents a user should not see, retrieval will surface them. Access control must be applied at retrieval time and filtered per user, never handled by instructing the model to withhold things. Query logs are also sensitive, since the questions asked can reveal as much as the documents.
When should you not use RAG?
When your content fits in the model's context window and rarely changes — just include it and skip an entire retrieval system. When questions require computation, aggregation, or joins, use a database query. And when the corpus is small, conventional search with good filters often serves users better than a generated answer.
