Home / Fundamentals / Post-training & Alignment
How does supervised fine-tuning turn a base model into an assistant? Discuss chat templates, loss masking and data.
30-second answer
SFT trains on conversations formatted with a chat template whose special tokens mark system, user, assistant and tool turns, using the same next-token loss as pre-training but masked so only assistant tokens contribute. The template must match inference exactly. A modest amount of high-quality, diverse data teaches format and behaviour, since knowledge comes from pre-training; coverage of skills such as code, math, tools, refusals and multilingual text matters more than raw volume. Use low learning rates for 1–3 epochs, evaluate on tasks, mix in general data to limit forgetting, and avoid training the model to assert facts it doesn't know.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
0–5 min Clarify and frame
Ask: base model size, target use (general assistant or domain), available data and labelers, compute. Framing: "I'll explain what SFT changes and why it works with little data, the mechanics (template, masking, packing), data strategy, hyperparameters, and how to evaluate and avoid forgetting."
5–15 min Core explanation
A base model has learned language and knowledge but continues text: given a question, it might produce more questions. SFT teaches it the assistant behaviour: respond to the user, follow instructions, use a consistent format, stop at the end of the turn.
Chat template. Conversations are serialized with special tokens:
<|system|>You are a helpful assistant.<|end|>
<|user|>Summarize this email: …<|end|>
<|assistant|>The sender asks to move Friday's meeting…<|end|>
Inference must use exactly the same template; mismatches (missing tokens, different whitespace) measurably hurt quality.
Loss masking. Same cross-entropy as pre-training, but only on assistant tokens (including the end-of-turn token, so the model learns to stop):
labels = input_ids.clone()
labels[~assistant_mask] = -100 # system/user/tool tokens ignored
logits = model(input_ids).logits
loss = F.cross_entropy(logits[:, :-1].reshape(-1, V), labels[:, 1:].reshape(-1), ignore_index=-100)
Training on user tokens too teaches the model to imitate users, and wastes capacity.
15–25 min Deep dive: data strategy
Why little data works: SFT mostly selects a behaviour already latent in the base model (the "superficial alignment" view; e.g. LIMA with ~1,000 curated examples). But capabilities such as tool use, structured output or a new language need many more examples.
Data mix (example for a general assistant):
| Category | Purpose |
|---|---|
| General instruction following, writing | Core helpfulness, format |
| Coding, math with worked solutions | Reasoning style |
| Multi-turn conversations | Context use, follow-ups |
| Tool and function calling | Agentic use |
| Refusals and borderline requests | Safety policy (and avoiding over-refusal) |
| "I don't know" / clarification examples | Honesty |
| Long-context tasks | Use of long inputs |
| Multilingual | Coverage |
Sources: human-written (expensive, high quality), model-generated with human editing, distillation from a stronger model (check licences), rejection sampling from your own model (sample n, keep the best by verifier or reward model), and synthetic data with programmatic checks.
Quality controls: dedupe, decontaminate against evals, filter with a quality classifier or LLM judge, balance categories, and check for factual claims the model can't support.
25–32 min Hyperparameters, packing and forgetting
- Full fine-tuning (7B): LR ~1e-5 to 2e-5, 1–3 epochs, warmup + cosine; batch of ~64–128 conversations. LoRA: LR ~1e-4 to 2e-4.
- Packing: concatenate conversations up to the max length; use attention masks (or position resets) so conversations don't attend across boundaries.
- Loss normalization: averaging per token vs per sequence weights long responses differently; be deliberate.
- Forgetting: narrow SFT degrades general skills and safety. Mix in general data, keep LR low, evaluate broadly (capabilities and safety) before and after.
32–37 min Production example
A team fine-tuned on 50k customer-support transcripts (agent turns as targets). The model learned the company's tone but also the agents' habits: promising callbacks it couldn't make and quoting outdated policies. Filtering transcripts, rewriting answers to current policy, adding "I'll connect you to a human" examples and mixing in 20% general instruction data fixed the behaviour and restored general-task scores.
37–40 min Likely follow-ups
- SFT vs continued pre-training? Continued pre-training on raw domain text adds knowledge; SFT teaches behaviour. Domain adaptation often uses both.
- Can SFT increase hallucination? Yes, if it trains the model to answer questions whose answers it didn't know from pre-training.
- Why not stop at SFT? Preference optimization and RL improve on what demonstrations alone can teach, and are easier to label at scale.
What interviewers listen for
- Loss masking and template details.
- A data strategy beyond "collect lots of data".
- Awareness of forgetting and hallucination effects.
Red flags
- Training on all tokens including prompts by default.
- Treating SFT as the way to teach the model new facts.