Choosing between RAG and fine-tuning

“Should we fine-tune?” is usually asked too early. The useful question is what kind of gap you are trying to close, because retrieval and fine-tuning close very different ones.

Retrieval fixes knowledge gaps

If the model gives a wrong answer because it has never seen your contract terms, your product catalogue, or last quarter’s numbers, that is a knowledge gap. Retrieval-augmented generation solves it by putting the right documents in front of the model at the moment of asking.

It is the right default: your content stays where it lives, updates take effect immediately, and you can show a user exactly which source produced an answer.

Fine-tuning fixes behaviour gaps

If the model knows the material but consistently formats it wrong, adopts the wrong register, or ignores a structure you need every single time, that is a behaviour gap. Fine-tuning teaches a pattern that prompting keeps failing to enforce reliably.

The cost is real: you need curated examples, a retraining path for when requirements change, and a plan for the day your base model is deprecated.

Where teams pick wrong

The common mistake is reaching for fine-tuning to fix hallucinations about company facts. It rarely works — the model becomes more fluent in your style while remaining just as capable of inventing a policy number. The fix is grounding, not tone.

The reverse mistake is stuffing ever more instructions into a prompt to enforce a rigid output format, growing latency and cost with every iteration, when a small fine-tune would settle it permanently.

Most production systems we build end up using retrieval for knowledge and careful prompt design for behaviour, and reach for fine-tuning only once a specific, measured behaviour gap refuses to close.