Home / Fundamentals / Post-training & Alignment

Post-training & AlignmentStaffQuestion 7 of 7

Design the post-training pipeline that turns a new base model into a helpful, safe assistant with strong reasoning.

30-second answer

Define target capabilities, safety policy and an evaluation suite first. Stage 1: SFT on curated, diverse conversations (human, distilled and rejection-sampled), with loss masking and a fixed chat template. Stage 2: preference optimization (reward model plus RL, or online/iterative DPO) on fresh policy samples for helpfulness, honesty and harmlessness, with a KL anchor. Stage 3: RL with verifiable rewards for math, code and tool use (GRPO-style), with curriculum and anti-hacking checks. Interleave safety data and red-teaming, gate each stage on regression evals, and iterate the data flywheel from evaluation failures.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Clarify: base model size and quality, target product (general assistant with tools and reasoning), languages, safety policy, data and compute budgets, release timeline. Framing: "Start from evaluations and policy, then the stages, then the data flywheel, then gating and safety."

5–15 min Requirements and evaluation first

Capabilities: instruction following, multi-turn chat, coding, math and reasoning, tool use, long context, multilingual.

Behaviour: honesty (calibrated, admits uncertainty), harmlessness per written policy, low over-refusal.

Evaluation suite, defined before training:

  • Capability benchmarks (held out, decontaminated) plus internal task sets.
  • Human and LLM-judge preference evaluations (length-controlled).
  • Safety: harmful-request refusal, over-refusal on benign borderline requests, jailbreak robustness.
  • Regression set for each stage.

15–25 min The stages

Scroll sideways to see the whole diagram.

Post-training pipelineComponents: Base model (pre-trained); SFT (chat template · masked loss); Preference stage (RM + RL or online DPO); Reasoning RL (verifiable rewards · GRPO); Release gate (no regressions); Conversations (curated · distilled); Preference pairs (human + AI feedback); Verifiers (tests · answer checks); Evals + red-teaming (capability · safety · over-refusal); Data flywheel (failures → new prompts and pairs). Flow: Base model to SFT; SFT to Preference stage; Preference stage to Reasoning RL; Reasoning RL to Release gate; Conversations to SFT; Preference pairs to Preference stage; Verifiers to Reasoning RL; Evals + red-teaming to Release gate; Evals + red-teaming to Data flywheel (failures); Data flywheel to Conversations; Data flywheel to Preference pairs.Base modelpre-trainedSFTchat template· masked lossPreference stageRM + RL or online DPOReasoning RLverifiablerewards · GRPORelease gateno regressionsConversationscurated · distilledPreference pairshuman + AI feedbackVerifierstests · answer checksEvals + red-teamingcapability · safety· over-refusalData flywheelfailures → newprompts and pairsfailures
Post-training pipeline. The base model is fine-tuned on curated conversations, then optimized on preferences with a KL anchor, then trained with reinforcement learning on verifiable tasks. Every stage is gated on a fixed evaluation and red-teaming suite, and each failure found becomes new data for the next iteration.
  1. SFT. Curated conversations covering every capability, with a fixed chat template and loss masking. Sources: expert-written, distilled, rejection-sampled from the base model with verifiers.
  2. Preference optimization. Sample responses from the SFT model, collect human and AI-feedback comparisons for helpfulness, honesty and harmlessness, train a reward model, then RL (PPO) or online/iterative DPO with a KL anchor. Several rounds, each on fresh samples.
  3. Reasoning RL with verifiable rewards. Math, code and tool-use environments; GRPO-style training with difficulty curricula; anti-hacking checks; length management.
  4. Safety throughout. Safety data in SFT and preferences, red-teaming each checkpoint, targeted fixes. Avoid a separate late "safety fine-tune" that causes over-refusal.
  5. Final polish. A short preference round to restore conversational quality after reasoning RL; system-prompt and tool-format robustness.

25–32 min Data flywheel and gating

  • Every evaluation failure becomes data: targeted prompts, new preference pairs, new RL environments.
  • Rejection sampling from the latest model produces better SFT data for the next iteration (expert iteration).
  • Model merging/averaging of checkpoints from different stages or data mixes can combine strengths.
  • Gates: each stage must not regress the suite beyond thresholds; human review of sampled outputs; safety sign-off.
  • Track data provenance, decontamination and licences.

32–37 min Trade-offs and production example

Decision Trade-off
Human vs AI preference labels Quality and nuance vs scale and cost; judge bias
DPO vs RL Simplicity vs flexibility and on-policy learning
Heavy reasoning RL Better math/code vs longer outputs and chat-quality drift
Strict safety Fewer harmful outputs vs over-refusal

Example: after reasoning RL, a team's model gained 12 points on math but became verbose in chat and refused more borderline-benign requests. A final mixed preference round (chat helpfulness plus over-refusal pairs) and a reasoning-length budget for simple queries recovered chat quality without losing the math gains.

37–40 min Likely follow-ups

  • How many SFT examples? Hundreds of thousands to millions for broad capabilities; smaller curated sets for style.
  • How do you prevent benchmark contamination? Decontaminate all stages' data; keep private held-out sets.
  • How long does a cycle take? Weeks per iteration; the bottleneck is usually data and evaluation, not compute.

What interviewers listen for

  • Evaluation and policy defined first.
  • Coherent stages with reasons and order.
  • A data flywheel and gating.

Red flags

  • "SFT, then RLHF, done."
  • No safety or regression testing.