How I design enterprise RAG

RAG — retrieval-augmented generation — is the default pattern for putting a language model to work on your own knowledge, and for good reason. But the version that dazzles in a demo and the version that holds up across a real enterprise knowledge base are not the same system. The gap between them is architecture.

Retrieval quality is the whole game

A RAG system is only as good as what it retrieves. If the right passage never makes it into the context, no model can save the answer. So most of my design effort goes into retrieval, not generation: how documents are chunked, how they're embedded, how queries are expanded, and how results are re-ranked before they ever reach the model. Garbage retrieval, confident garbage answer.

Chunking is a design decision, not a default

How you split documents quietly determines everything downstream. Too large and you bury the signal; too small and you sever the context a passage needs to make sense. The right strategy depends on the documents — contracts, tickets, and wikis each want different treatment. I treat chunking as a first-class design choice and test it, rather than accepting a library default.

Ground every answer, and prove it

In an enterprise, an answer without a source is a liability. Every response should cite the documents it came from, so a user — and an auditor — can verify it. This isn't just trust theatre; visible citations are how you catch retrieval failures and how you make the system safe to rely on in a regulated setting.

Plan for the questions retrieval can't answer

Real users ask things the knowledge base doesn't contain. A production RAG system needs to know when to say “I don't have that” instead of inventing a plausible answer. Designing for graceful failure — confidence thresholds, fallbacks, honest non-answers — is what separates a system people trust from one they quietly stop using.

Evaluation is not optional

You cannot improve what you don't measure. Before launch I build an evaluation set of real questions with known-good answers, so every change to chunking, embeddings, or prompts can be scored rather than guessed at. RAG without evaluation is just vibes with a vector database attached.

If you're choosing between this and customising the model itself, read the companion piece on RAG versus fine-tuning — it's the decision that most often gets made for the wrong reasons.