Out of the box, a large language model knows a lot about the world and nothing about your products, policies or customers. There are two main ways to change that: retrieval-augmented generation (RAG) and fine-tuning. They are often presented as alternatives, but they solve different problems.
RAG: give the model the right facts at the right time
With RAG, your documents are split into passages, converted into embeddings and stored in a search index. When a question arrives, the system retrieves the most relevant passages and passes them to the model along with the question.
- Best for: answering questions from knowledge that changes, such as help articles, policies, product data and contracts.
- Strengths: answers can cite their sources, updates are as simple as re-indexing a document, and access controls can follow your existing permissions.
- Watch out for: poor chunking, outdated documents and retrieval that finds similar-sounding but wrong passages.
Fine-tuning: teach the model a behaviour
Fine-tuning continues training a model on your own examples, so it learns a format, tone or task.
- Best for: consistent output formats, a specific writing style, classification and extraction tasks, or making a smaller model perform like a larger one on a narrow job.
- Strengths: shorter prompts, more consistent results and potentially lower cost per request.
- Watch out for: it is a poor way to store facts, which go stale and cannot be cited, and every change means preparing data and training again.
A simple rule
Use RAG for what the model should know. Use fine-tuning for how the model should behave.
When to combine them
Mature systems often use both: RAG supplies current, permission-aware facts, while a fine-tuned model formats answers consistently or handles a specialised task such as extracting fields from invoices.
Where to start
- Start with a strong base model, good prompts and RAG. This solves most business use cases.
- Build an evaluation set of real questions with expected answers before you change anything.
- Improve retrieval first: better document structure, metadata filters and re-ranking often beat a bigger model.
- Consider fine-tuning only when evaluations show a consistent behaviour gap that prompting cannot close.
Whichever approach you choose, treat evaluation as part of the product. Without it, you cannot tell whether a change made things better or worse.
