Almost every private AI enquiry we receive arrives phrased the same way: we want to train a model on our data. In most cases that is not what the project needs, and acting on it directly produces a system that is more expensive to run, harder to maintain, and worse at the job than the alternative.
The distinction is simple to state and consistently mis-applied.
What each one actually changes
Retrieval-augmented generation (RAG) changes what the model can see when it answers. Your documents are indexed; at question time the relevant passages are fetched and placed in front of the model; the model answers from them. The model's weights are untouched.
Fine-tuning changes how the model behaves. You continue training it on examples so it adopts a format, a register, a task specialisation, or fluency in a domain or language that was thinly represented in its original training data.
The sentence worth remembering: retrieval changes what the model knows about; fine-tuning changes how the model acts.
Why "train it on our documents" usually means retrieval
When a team says they want a model trained on their documents, what they almost always want is for it to answer questions about those documents accurately. Fine-tuning is a poor mechanism for that, for four reasons.
Facts do not reliably stick. Training a model on a document does not install its contents as retrievable facts. It shifts the statistical tendencies of the model's output. Ask it a specific question and you may get something that resembles the document without being it — which is the worst possible failure mode, because it is fluent and wrong.
No provenance. A fine-tuned model cannot tell you which document an answer came from, because the answer did not come from a document; it came from adjusted weights. In regulated work, an answer nobody can trace to a source is frequently unusable regardless of whether it happens to be correct.
Your documents change. Every new contract, report or policy revision means either retraining or a model that is quietly out of date. With retrieval, new material is answerable as soon as it is indexed.
Access control becomes impossible. This is the one that ends the conversation in most enterprises. A fine-tuned model has absorbed everything it was trained on into a single set of weights. It cannot show one answer to a manager and a different one to a contractor. With retrieval, permissions are applied when passages are fetched, so a user only ever gets answers built from documents they were already entitled to open.
That last point is not a tuning problem. It is structural, and no amount of prompting fixes it.
Where fine-tuning genuinely earns its cost
None of which makes fine-tuning useless. It is the right tool for a specific and real set of problems, all of which are about behaviour rather than knowledge:
- Output format. When responses must be valid structured data or follow a template a downstream system parses, training the behaviour is far more reliable than repeating instructions in every prompt.
- Domain vocabulary. Clinical shorthand, legal drafting conventions, engineering part nomenclature — language patterns that are constant in your work and largely absent from public training data.
- Languages and scripts with thin coverage. Where a model's grasp of a language, dialect or script is weak, no amount of retrieval fixes it. The gap is in the model, so the model is what has to change.
- Task specialisation for smaller models. A 7B model fine-tuned for one narrow task can match a much larger general model at that task, at a fraction of the serving cost. Since inference is paid on every request and training is paid once, this arithmetic often favours fine-tuning strongly.
- Consistent policy behaviour. Where a model must reliably refuse, escalate or flag in defined ways, that can be trained rather than hoped for.
The comparison, side by side
| Retrieval (RAG) | Fine-tuning | |
|---|---|---|
| Changes | What the model sees | How the model behaves |
| New information | Available once indexed | Requires retraining |
| Citations | Every answer names its source | No inherent provenance |
| Access control | Enforced at retrieval time | Not possible after training |
| Cost profile | Build and storage | GPU time, repeated each update |
| Iteration speed | Minutes | Hours to days |
| Best for | Facts, documents, anything that changes | Format, tone, specialisation, thin domains |
They are not mutually exclusive
The framing as a choice is itself part of the problem. Mature private deployments frequently use both: retrieval supplies the facts and the citations, while a light fine-tune fixes the output format or the domain vocabulary that retrieval alone cannot correct.
The order matters, though, and it is almost always the same:
- Build retrieval first. It is cheaper, faster to iterate, and it will expose what the real failure modes are.
- Measure. With an evaluation set drawn from your actual questions, graded by the people who will use the system.
- Fine-tune only what retrieval could not fix. By this point you know precisely what that is, which also tells you what training data you need to assemble.
Teams that invert this order spend GPU budget solving problems that better chunking would have solved for free.
Before either: check whether retrieval is even the bottleneck
A large share of disappointing private assistants are not failing because the model is too small or insufficiently trained. They are failing because the right passage never reached the model.
Common causes, all cheaper to fix than either technique above:
- Chunking that ignores document structure. Contracts, manuals and reports break into passages differently. A single chunk size applied to all of them loses the thread.
- Vector search alone. Pure semantic search misses exact identifiers — clause numbers, part codes, policy references. Hybrid keyword-plus-vector search with a reranking pass typically produces a larger quality improvement than moving to a bigger model, and it does not raise your hardware bill.
- No evaluation set. Without one, every change is an opinion. With one, you can tell whether a change helped.
We cover how this fits together in building a private knowledge assistant.
How AxcelerateAI Helps
We build both, and we will tell you which one your project needs:
- Retrieval pipelines with hybrid search, reranking and citations, running entirely on your infrastructure.
- Fine-tuning on open-weight models, on your own hardware, with the weights handed over to you.
- An honest recommendation first — including when the answer is that better retrieval makes fine-tuning unnecessary.
