RAG explained: how retrieval-augmented generation actually works

What RAG is, and what it is not

Retrieval-augmented generation (RAG) is a simple idea with a lot of engineering underneath it. Instead of asking a language model to answer from whatever it absorbed in training, you first fetch the passages from your own documents that are most relevant to the question, then hand those passages to the model alongside the question. The model writes the answer; your documents supply the facts.

That separation is the whole point. A large language model is a very good reader and writer and a very unreliable encyclopaedia. Its knowledge is frozen at training time, it has never seen your contracts or your support tickets, and when it does not know something it will often produce a plausible guess. RAG keeps the knowledge in a store you control and uses the model for the part it is genuinely good at: reading a few pages and composing a precise answer.

RAG is not a product you buy, and it is not a single API call. It is an architecture with two halves. The first half runs offline and turns documents into something searchable. The second half runs on every request and turns a question into a grounded answer. Most failures we are asked to fix sit in the first half, where nobody was looking.

The pipeline, step by step

The diagram below shows both halves. Read the top row first; it is the part that usually gets the least attention and causes the most trouble.

Index time

Ingest. Pull documents from wherever they live: SharePoint, Confluence, a ticketing system, a database. Extract clean text from PDFs, slides and HTML. Keep the metadata (owner, date, access group, product line) because you will filter on it later.

Chunk. Split each document into passages small enough to embed well and large enough to carry meaning. Chunking by heading structure usually beats fixed character windows. Store the chunk, its position and its parent document.

Embed. Run each chunk through an embedding model to produce a vector that captures its meaning. Similar passages end up close together in that vector space.

Store. Write the vectors, the chunk text and the metadata into a vector database. For most teams this is pgvector inside the PostgreSQL they already run; larger or latency-sensitive systems use a dedicated engine.

Query time

Retrieve. Embed the question with the same model, find the nearest chunks, and apply any metadata filters (only this customer's contracts, only documents the user may see). We nearly always combine vector search with keyword search, a pattern called hybrid search, because exact terms such as product codes and names embed poorly.

Rerank. Pass the top candidates through a reranker, a smaller model that scores each chunk against the question more carefully than the first-pass search can. Keep the best few.

Generate. Build a prompt from your instructions, the retrieved chunks and the question. Ask the model to answer only from the supplied material and to cite which chunk supports each claim. Return the answer with those citations so the user can check.

Why retrieval beats fine-tuning for knowledge

Teams new to LLMs often assume the way to teach a model about their business is to train it on their documents. In practice fine-tuning changes how a model behaves far more reliably than what it knows, and it is a poor fit for information that changes weekly.

The two are not rivals. A mature system may use a fine-tuned model for format and a retrieval layer for facts. But the order matters, and we cover it in prompt engineering vs fine-tuning: prompt first, retrieve second, fine-tune last.

Where RAG goes wrong

Almost every underperforming RAG system we audit fails in one of the same handful of ways. None of them is about the language model.

The common thread is that these are data-pipeline problems. They are found by looking at what the retriever returns for real questions, not by swapping models.

Five design decisions that matter

1. Chunking strategy

Start with structure-aware chunking: split on headings, keep tables intact, and attach the document title and section path to every chunk so it still makes sense out of context. Overlap adjacent chunks slightly so a sentence that straddles a boundary survives. Tune size by looking at retrieval results, not by guessing.

2. Embedding model

The embedding model decides what "similar" means. Choose one that handles your languages and your domain vocabulary, and remember that changing it later means re-embedding everything. Our guide on choosing embedding models covers the criteria; the embeddings strategy service covers doing it for you.

3. Hybrid retrieval with filters

Combine vector similarity with keyword matching and merge the two result lists. Add metadata filters for tenant, access group, date and document type. This single change fixes more "it can't find the obvious document" complaints than any other.

4. Reranking and context budget

Retrieve generously, then rerank down to a handful. Stuffing twenty chunks into the context window costs money and makes the model's job harder. Three to six well-chosen passages usually beat twenty mediocre ones.

5. Grounding and citations

Instruct the model to answer only from the supplied passages, to say "I could not find this in the documentation" when they do not cover the question, and to cite the passage for each claim. Show those citations in the interface. Users forgive a cautious answer; they do not forgive a confident wrong one.

RAG is an attack surface

Every document you index is text that will be placed into a prompt. If an attacker can get text into your document store, they can get instructions in front of your model. This is indirect prompt injection, and it is the defining security problem of RAG.

The second problem is leakage. If the retriever ignores access control, a junior employee can ask the assistant what the board discussed last quarter, and it will answer. Permissions must be enforced at retrieval time, using the same identity the user logged in with, not reconstructed by the model.

The practical controls are well understood: sanitise and label ingested content, filter by permission inside the query, treat retrieved text as data rather than instructions, limit what the model can do with tools, and log every retrieval. We go through them in the LLM security checklist, and our secure RAG architecture and prompt injection defence services implement them.

How to evaluate a RAG system

You cannot improve what you only test by hand. Build a golden set early: fifty to two hundred real questions with known correct answers and the document passages that support them. Then measure two things separately.

• Retrieval quality. For each question, did the right passages appear in the top results? If not, nothing downstream can fix it.
• Answer quality. Given the right passages, was the answer correct, complete and faithful to the source? Model-graded rubrics work well here, with a human spot check.

Run the set on every change to chunking, embeddings, prompts or model. Track the numbers over time. This is the difference between tuning and guessing, and it is the subject of evaluating LLM outputs and our fine-tuning and evals service.

When RAG is the wrong tool

RAG answers questions from documents. It does not do arithmetic across a database, follow a multi-step process or take actions. If the question is "how many open invoices over 60 days do we have", you want a query against the finance system, possibly generated by the model, not a search over PDFs. If the task is "book the meeting and update the CRM", you want an agent with tools, as described in agentic workflows 101. And if the whole corpus is a few dozen pages, you may not need retrieval at all; put it in the prompt.

The best RAG systems are often narrow. A support copilot over one product's documentation, with clean sources and a maintained evaluation set, beats an "ask anything about the company" assistant that nobody trusts.

Key takeaways

• RAG keeps knowledge in a store you control and uses the model only to read and write.
• Most failures are in ingestion, chunking and retrieval, not in the language model.
• Use hybrid search with metadata filters; vector-only retrieval misses exact terms.
• Retrieve broadly, rerank tightly, and give the model a refusal path.
• Enforce access control at retrieval time and treat retrieved text as untrusted.
• Build a golden set before you tune anything, and measure retrieval and answers separately.

Related services