Home / Fundamentals / Post-training & Alignment

Supervised fine-tuning

How a base model becomes an assistant: chat templates, loss masking, data quality over quantity, packing, hyperparameters, forgetting, and where SFT sits in the post-training pipeline.

12 min read

A pre-trained base model continues text; it doesn't reliably follow instructions, hold a conversation or stop when it should. Post-training turns it into an assistant, usually in stages.

Scroll sideways to see the whole diagram.

Post-training pipelineComponents: Base model (pre-trained); SFT (chat template · masked loss); Preference stage (RM + RL or online DPO); Reasoning RL (verifiable rewards · GRPO); Release gate (no regressions); Conversations (curated · distilled); Preference pairs (human + AI feedback); Verifiers (tests · answer checks); Evals + red-teaming (capability · safety · over-refusal); Data flywheel (failures → new prompts and pairs). Flow: Base model to SFT; SFT to Preference stage; Preference stage to Reasoning RL; Reasoning RL to Release gate; Conversations to SFT; Preference pairs to Preference stage; Verifiers to Reasoning RL; Evals + red-teaming to Release gate; Evals + red-teaming to Data flywheel (failures); Data flywheel to Conversations; Data flywheel to Preference pairs.Base modelpre-trainedSFTchat template· masked lossPreference stageRM + RL or online DPOReasoning RLverifiablerewards · GRPORelease gateno regressionsConversationscurated · distilledPreference pairshuman + AI feedbackVerifierstests · answer checksEvals + red-teamingcapability · safety· over-refusalData flywheelfailures → newprompts and pairsfailures
Post-training pipeline. The base model is fine-tuned on curated conversations, then optimized on preferences with a KL anchor, then trained with reinforcement learning on verifiable tasks. Every stage is gated on a fixed evaluation and red-teaming suite, and each failure found becomes new data for the next iteration.

What SFT does

Supervised fine-tuning trains on (prompt, ideal response) conversations with the same next-token loss as pre-training, but:

  • Formatted with a chat template: special tokens mark system, user, assistant and tool turns, e.g. <|user|> … <|assistant|> … <|end|>. The template used in training must match the one used at inference exactly.
  • Loss on assistant tokens only: prompt tokens are masked out (label −100), so the model learns to produce responses, not to imitate users. Multi-turn data computes loss on every assistant turn.
labels = input_ids.clone()
labels[~is_assistant_token] = -100      # ignored by cross-entropy
loss = F.cross_entropy(logits[:, :-1].flatten(0, 1), labels[:, 1:].flatten())

Data matters more than quantity

  • A small set of very high-quality, diverse examples can produce a strong assistant style (the LIMA work used ~1,000), because SFT mostly teaches format and behaviour on top of knowledge already learned in pre-training.
  • Capabilities need breadth: coding, math, tool use, multilingual, refusals, long-context tasks, each with enough examples.
  • Sources: human-written, model-generated then human-edited, distilled from stronger models (check terms of use), and rejection-sampled from your own model (generate many, keep the best by a verifier or reward model).
  • Avoid teaching hallucination: training the model to state facts it doesn't know encourages confident fabrication. Include examples that say "I don't know" when appropriate.

Mechanics and hyperparameters

  • Full fine-tuning of a 7B model typically uses learning rates around 1e-5 to 2e-5, 1–3 epochs, cosine decay with warmup. LoRA uses higher rates (around 1e-4 to 2e-4).
  • Packing several short conversations into one sequence improves throughput; mask attention across conversation boundaries if possible.
  • Monitor held-out loss and, more importantly, task evaluations; SFT overfits fast.

Forgetting

Fine-tuning on a narrow distribution degrades other abilities. Mitigations: mix in general instruction and pre-training data, lower learning rates, fewer epochs, LoRA (which changes fewer weights), and evaluate broadly before and after.

What to remember

  • Same loss as pre-training, masked to assistant tokens, with an exact chat template.
  • Quality and coverage beat volume.
  • SFT teaches format and behaviour; preference optimization and RL refine it.