Home / Fundamentals / Post-training & Alignment

Post-training & AlignmentSeniorQuestion 6 of 7

How are reasoning models trained with reinforcement learning? Explain RL with verifiable rewards and GRPO.

30-second answer

Use tasks with programmatically checkable outcomes: math answers compared with references, code run against tests, format and constraint checks. For each prompt, GRPO samples a group of G responses, scores them, and gives every token in response i the advantage (r_i − mean)/std, then applies a PPO-style clipped update with a KL penalty; the group mean replaces a learned value model. Prompts all-correct or all-wrong give no signal, so difficulty is curated. Longer reasoning with self-checking emerges, accuracy scales with thinking tokens, and risks include reward hacking of weak verifiers, length growth and entropy collapse.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll describe the RLVR setup, derive GRPO from the policy gradient with a baseline, discuss data and curriculum, the dynamics you observe (longer reasoning), and failure modes."

5–15 min Core explanation

Setup: prompts with automatically checkable outcomes.

  • Math with reference answers (normalize and compare, or symbolic equivalence).
  • Code with unit tests (run in a sandbox with timeouts).
  • Instruction constraints ("answer in under 50 words, in JSON").

Reward is typically 1/0 for correctness, sometimes plus small format terms.

Policy gradient with a baseline:

∇J = E[ (R − b) · ∇ log π(y|x) ]

The baseline b reduces variance. PPO learns b with a value model; GRPO estimates it from a group of samples for the same prompt.

GRPO:

  1. For prompt x, sample G responses y_1..y_G from the current policy.
  2. Compute rewards r_i.
  3. Advantage for all tokens of y_i: A_i = (r_i − mean(r)) / std(r).
  4. Maximize the PPO-clipped objective averaged over tokens, minus β · KL(π ‖ π_ref).

Scroll sideways to see the whole diagram.

One GRPO update for a promptTrainer → Policy: prompt x: sample G = 8; Policy → Trainer: 8 reasoning traces + answers; Trainer → Verifier: check each answer; Verifier → Trainer: rewards: 1 0 0 1 1 0 0 0; Note: mean 0.375, std 0.48 → A = +1.29 (correct), −0.77 (wrong); Trainer → Reference: token log-probs; Reference → Trainer: for the KL penalty; Trainer: clipped policy-gradient step − β·KL; Trainer → Policy: sync updated weights; [if all 8 rewards are equal]; Note: every advantage is 0: no learning signal, so filter prompts by pass rate.TrainerPolicyrollout engineVerifiertests · answer checkReferencefrozenprompt x: sample G = 88 reasoning traces + answerscheck each answerrewards: 1 0 0 1 1 0 0 0mean 0.375, std 0.48 → A = +1.29 (correct), −0.77 (wrong)token log-probsfor the KL penaltyclipped policy-gradient step − β·KLsync updated weightsIF ALL 8 REWARDS ARE EQUALevery advantage is 0: no learning signal, so filter prompts by pass rate
One GRPO update for a prompt. The policy samples a group of eight responses. A programmatic verifier scores them; three are correct. Each response's advantage is its reward minus the group mean, divided by the group's standard deviation, so correct responses are pushed up and wrong ones down without a learned value model. A KL term to the frozen reference keeps the policy anchored.

15–25 min Deep dive

Why groups work: with binary rewards and G = 16, a prompt the model solves 4/16 times gives correct responses a positive advantage and incorrect ones a negative advantage, pushing probability toward successful reasoning paths. No critic needs to be trained (memory and stability benefits).

Zero-signal prompts: if all 16 responses are correct (or all wrong), all advantages are zero. So:

  • Filter or curriculum: keep prompts with pass rates between ~10% and ~90% for the current model, re-estimated over training.
  • Dynamic sampling: oversample and drop zero-variance groups.

Normalization biases: averaging the loss per response divides each token's gradient by response length, which reduces the penalty on long wrong answers; dividing by group std up-weights prompts with low variance. Variants (e.g. token-level averaging, removing std normalization) address these.

Dynamics: response length and reflective behaviour ("wait, let me check") tend to grow during training, and accuracy improves with more reasoning tokens at test time. Watch policy entropy: collapse means exploration stops; asymmetric clipping (allowing larger increases for low-probability tokens) and temperature help.

Infrastructure: generation dominates cost (long samples, G per prompt), so RL systems separate fast inference engines for rollouts from training, with weight syncing and off-policy corrections.

25–32 min Failure modes

Failure Example Mitigation
Weak verifier Passing tests by special-casing inputs Hidden tests, more tests, property checks
Parser exploits Multiple answers in output, one matches Strict answer extraction
Environment tampering (agents) Editing tests Read-only test files, monitoring
Length explosion Overthinking easy problems Length penalties, budgets
Language mixing / unreadable chains Mixed-language reasoning Language consistency reward, SFT warm start
Narrow gains Better math, worse chat Mix tasks, keep general RL/preference stages

A short SFT "cold start" on readable reasoning traces before RL improves stability and readability.

32–37 min Production example

A team ran GRPO on a 7B model with 40k math problems, G = 16. Accuracy on their held-out set rose from 48% to 63%, but after ~1,000 steps, 30% of training prompts had zero-variance groups and progress stalled. Re-filtering prompts by current pass rate each epoch and adding harder problems restarted progress (to 69%). They also found a parser bug rewarding responses that listed several candidate answers; fixing it caused a temporary dip but removed the behaviour.

37–40 min Likely follow-ups

  • Why not PPO with a value model? It works too; value models for long reasoning are hard to train and cost memory. GRPO is simpler.
  • Process rewards? Step-level rewards give denser signal but are costly and hackable; outcome rewards with many samples work surprisingly well.
  • Distillation vs RL for small models? Distilling traces from a strong reasoner is cheaper and often better than RL on the small model alone.

What interviewers listen for

  • Group-relative advantage and its connection to baselines.
  • Difficulty filtering and zero-variance groups.
  • Concrete failure modes with mitigations.

Red flags

  • Thinking GRPO needs a reward model for math.
  • Ignoring verifier quality.