RAG or fine-tuning? Making the right call in enterprise AI projects
This question comes up in almost every first meeting, and it is usually framed the wrong way. RAG and fine-tuning are not alternatives to each other; they are two tools that solve different problems. The confusion comes from both being described as "running the model on our own data".
What is RAG and when should you use it?
Retrieval-Augmented Generation (RAG) means giving the model the relevant documents together with the question. The model builds its answer from the text in front of it, not from its own memory. Technically it is three steps: turn the question into a vector, find the closest documents, and pass those documents to the model with the question.
Use RAG when your information changes over time, when you need to show where an answer came from, when different users may see different documents, or when you have a large volume of documents. If even one of these is true, RAG is the right answer, and almost every enterprise scenario has at least one.
What is fine-tuning and when should you use it?
Fine-tuning retrains a model on example input and output pairs to shift its behaviour. You are not telling the model "answer this question like this"; you are telling it "answer questions of this kind in this shape". What it learns is a pattern, not knowledge.
Use fine-tuning when the output has to follow a very specific format, when your organisation has its own language or terminology, when your prompt has grown so long that you want to cut its cost, or when you are trying to bring a small model close to large model quality on one specific task.
What is the difference between the two?
| Criterion | RAG | Fine-tuning |
|---|---|---|
| What it carries | Knowledge | Behaviour and format |
| Updating knowledge | Change the document, effective immediately | Needs retraining |
| Citing sources | Possible by design | Not possible |
| Access control | Can be filtered per user | The model is the same for everyone |
| What sets the timeline | Document quality and access structure | Data preparation and training rounds |
| Main cost item | Tokens per query and vector infrastructure | Data preparation and training rounds |
| Hallucination risk | Can be reduced (answers tied to sources) | Unchanged, can even increase |
Why do so many projects choose the wrong one?
Fine-tuning sounds more serious. "We trained our own model" lands better in a board meeting than "we search the documents and hand them to the model". That is a technical decision made for a non-technical reason, and it gets expensive.
The second reason is a misconception: "We want the model to know our company." Fine-tuning does not do that. When you give the model your company documents as training data, it learns their style, not a reliable memory of their content. Ask it a question and it produces confident, wrong answers, which is worse than no answer at all.
What is the right order?
- Prompt engineering first. Some problems are solved by a well written system prompt. Do not move to the expensive option before trying the cheapest one.
- Then RAG. If the task needs access to knowledge, start here and measure accuracy.
- Look at the remaining errors. If they are "wrong information" errors, improve RAG: chunking strategy, retrieval quality, reranking. If they are "right information, wrong shape" errors, that is where fine-tuning comes in.
- Add fine-tuning last. And add it knowing that you will repeat this step with every model update.
Which mistakes happen in practice?
On the RAG side, the most common mistake is splitting documents into fixed length chunks. A clause cut from the middle of a contract cannot be found, and when it is found it means nothing. Chunking should follow the document's own structure: heading, clause, section.
The second common mistake is having no evaluation set. A RAG system without one cannot be measured; any claim of "we improved it" is just a feeling. A fixed set of 50 to 100 questions can be run after every change, and the improvement shows up as a number.
On the fine-tuning side, the most common mistake is training on too little, inconsistent data. Training on a few hundred examples usually does not improve the model; it only makes it less stable.
Three questions for the decision
When you choose, ask these three questions:
- Must the answer be grounded in a document? If yes, RAG.
- Does the information change more than once a month? If yes, RAG.
- Is the problem the shape of the answer rather than the knowledge? If yes, fine-tuning.
If the answers are mixed, start with RAG. Moving from RAG to fine-tuning is easy; going the other way means months spent for nothing.
Frequently asked questions
Can RAG and fine-tuning be used together?
Yes, and most mature systems use both. RAG carries the knowledge, fine-tuning adjusts the behaviour. The right order is to build and measure RAG first, then add fine-tuning if the remaining errors are about format.
Does fine-tuning teach a model new knowledge?
Not reliably. Fine-tuning shifts weights; it does not build a knowledge base. What it learns cannot be verified, cannot cite a source and needs retraining to update. The right way to carry knowledge is to put it in the context, which is what RAG does.
How long does it take to build RAG?
Document quality sets the timeline, not the technology. With an orderly, searchable and current archive, a basic RAG system starts working quickly. A production grade system, with an evaluation set, source citations, access control and monitoring, takes noticeably longer. We do not give a schedule before the scope is clear; once it is, we share the project timeline.
How do you measure RAG accuracy?
It takes two separate measurements: retrieval accuracy (was the right document found) and generation accuracy (was the right answer drawn from it). Most failed RAG projects never separate the two, so they cannot tell where it breaks. A fixed evaluation set of 50 to 100 questions is the cheapest way to make that split.
What does fine-tuning cost?
In most enterprise scenarios training itself costs less than people expect; the real cost is data preparation. A good fine-tuning dataset needs thousands of examples, and preparing them is human work. There is also a hidden cost: you repeat the training every time the base model is updated.
Enterprise AI
AI that runs on your ERP data and survives production.
Related notes
Note KVKK compliant AI architecture: where should the data live?
AI projects that handle personal data have three architectural options: mask the data before it reaches the model, use an enterprise cloud hosted in a European region, or run an open model on your own infrastructure. The choice is a trade-off between cost and risk, and which data goes where should be put in writing at the start of the project.