How RAG actually works

Retrieval-augmented generation sounds like jargon and is actually a simple, sensible idea. If you're approving an AI investment, you should understand it well enough to ask the right questions — so here it is without the mystique.

The problem it solves

A language model knows what it learned during training. It doesn't know your contracts, your policies, or anything that happened after its training cut-off, and when asked anyway, it will often invent a confident, wrong answer. RAG fixes this by giving the model the relevant information at the moment it answers.

The four steps

First, your documents are broken into passages and converted into numerical representations called embeddings, which capture meaning. These live in a vector database. Second, when a user asks a question, the question is embedded the same way. Third, the system finds the passages whose meaning is closest to the question — retrieval. Fourth, those passages are handed to the model along with the question, and it composes an answer grounded in them.

Why it's powerful

The model is no longer guessing from memory; it's reasoning over the exact documents you gave it. Update the documents and the answers update instantly — no retraining. And because you know which passages were retrieved, you can show the user the sources, which is what makes the whole thing trustworthy enough for enterprise use.

Where it goes wrong

Every weakness traces back to step three: if retrieval surfaces the wrong passages, the answer will be wrong no matter how capable the model is. That's why, in practice, building good RAG is mostly about building good retrieval — a point I go deeper on in how I design enterprise RAG.

That's the whole idea. Everything else — chunking strategies, re-rankers, embeddings — is craft in service of those four steps.