Home / Fundamentals / Post-training & Alignment
When would you fine-tune instead of using RAG or prompting, and vice versa?
30-second answer
Start with prompting and evaluation. Use RAG when answers need fresh, private, large or citable knowledge, or knowledge that changes, since retrieval is updatable and auditable. Fine-tune to change behaviour: consistent format and tone, a narrow skill, domain style, tool-use patterns, or to replace long prompts at high volume or distil a large model into a cheaper one. Fine-tuning is unreliable for injecting facts and can increase hallucination on unseen ones. They combine well: fine-tune for behaviour and retrieval use, retrieve for knowledge, and justify each step with evals.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
0–5 min Clarify and frame
Ask: what's wrong today (missing knowledge, wrong behaviour, cost, latency)? How often does the knowledge change? Is attribution required? Volume? Framing: "I'll separate knowledge problems from behaviour problems, give a decision process, then cover combinations and evaluation."
5–15 min Core explanation
| Need | Best first tool | Why |
|---|---|---|
| Better instructions, format, tone | Prompting | Fastest iteration, no training |
| Fresh, private, large or citable knowledge | RAG | Updatable, auditable, access-controlled |
| Consistent style or format at scale | Fine-tuning | Behaviour baked in; shorter prompts |
| A narrow skill the model does poorly | Fine-tuning (with evals) | Learns patterns from examples |
| Lower cost/latency at high volume | Fine-tune a smaller model (distillation) | Smaller model matches task quality |
| Tool-use reliability | Fine-tuning on tool traces | Learns formats and when to call |
Why fine-tuning is poor at adding facts: new facts seen a few times in fine-tuning are learned slowly and unreliably, and training on facts the model didn't already know can increase hallucination on other questions. Facts also can't be updated or deleted easily, and answers can't cite sources.
15–25 min Deep dive: a decision process
- Build an evaluation set of real tasks with graders.
- Prompting baseline: clear instructions, examples, structured output. Measure.
- Failures due to missing knowledge? Add retrieval; measure retrieval recall and answer faithfulness separately.
- Failures due to behaviour (format drift, tone, not following procedure, poor tool use) after good prompting? Fine-tune on examples; LoRA first.
- Cost/latency too high? Distil: fine-tune a smaller model on the large model's outputs for this task.
- Re-evaluate after each step; keep the simplest system that meets the bar.
Combinations:
- RAG + fine-tuning: fine-tune the model to use retrieved context well (cite, abstain when context is insufficient, ignore distractors), and retrieve the knowledge.
- Fine-tuned retriever/reranker: often the best investment in a RAG system.
- Continued pre-training on a large domain corpus (e.g. legal or medical text) adds domain fluency; still pair with RAG for specific facts.
25–32 min Trade-offs
| Factor | RAG | Fine-tuning |
|---|---|---|
| Knowledge freshness | Minutes (re-index) | Retrain |
| Attribution | Citations | None |
| Access control | Per-document filters | Not possible inside weights |
| Per-request cost | Longer prompts | Shorter prompts |
| Upfront effort | Ingestion pipeline | Data curation, training, evals |
| Failure mode | Retrieval misses | Forgetting, hallucination, stale behaviour |
32–37 min Production example
A legal team fine-tuned a model on 10,000 internal Q&A pairs to "teach it company policy". It answered confidently but cited superseded policies and invented clauses. They switched to RAG over the policy repository with citations, and fine-tuned only for answer structure and abstention. Accuracy on their eval rose from 61% to 88%, and policy updates took effect the same day.
37–40 min Likely follow-ups
- What about long-context models instead of RAG? For small, stable corpora, putting everything in context (with prompt caching) can work; RAG scales and filters better.
- How much data to fine-tune? Hundreds to a few thousand high-quality examples for behaviour with LoRA; more for new skills.
- How do you keep a fine-tune current? Version datasets, retrain on base-model upgrades, and run regression evals.
What interviewers listen for
- Knowledge vs behaviour framing.
- An evaluation-driven decision process.
- Combinations, and why fine-tuning facts is risky.
Red flags
- "Fine-tune on our documents so the model knows them."
- No mention of evaluation.