RAG vs fine-tuning: choosing the right tool

“Should we use RAG or fine-tune?” is the most common technical question I get, and it's usually framed as a battle. It isn't. They solve different problems, and the right answer is often “RAG now, and maybe a little fine-tuning later, if ever.”

What each one actually changes

RAG changes what the model knows by handing it relevant information at question time. Fine-tuning changes how the model behaves by adjusting its weights on examples. One injects knowledge; the other shapes style, format, or a narrow skill. Confusing the two is the root of most bad decisions here.

Reach for RAG when knowledge changes

If the problem is “the model needs to know our policies, our products, our documents” — and especially if those change — RAG is almost always the answer. You update a knowledge base, not a model. It's cheaper, faster to ship, auditable through citations, and it doesn't go stale the moment a document is revised.

Reach for fine-tuning when behaviour must change

Fine-tuning earns its keep when you need consistent tone, a rigid output format, or a specialised skill that prompting can't reliably produce — and you have good training examples. It is not a way to teach the model facts; that's a job it does badly and expensively. If your instinct is to fine-tune so the model “knows” your data, that's a sign you want RAG.

The honest default

Start with RAG and good prompting. It solves the majority of enterprise use cases, ships faster, and keeps your knowledge where you can govern it. Consider fine-tuning only once you've hit a specific, measured ceiling that retrieval and prompting genuinely can't clear. Most teams never need to. The ones that fine-tune first usually spend a lot of money to arrive back where they started.