Home / Fundamentals / Post-training & Alignment

Reinforcement learning for reasoning

Reinforcement learning with verifiable rewards, GRPO's group-relative advantages, reward hacking and length effects, test-time compute, and distilling reasoning into smaller models.

13 min read

Reasoning models are trained to produce long chains of thought before answering, mostly with reinforcement learning on tasks whose answers can be checked automatically.

Verifiable rewards (RLVR)

Instead of a learned reward model, use a program:

  • Math: compare the final answer with the reference (after normalization).
  • Code: run unit tests in a sandbox.
  • Formats, instruction constraints, some logic puzzles: rule-based checks.

Programmatic rewards are much harder to hack than a learned reward model, which allows long RL runs. They only cover checkable domains, so verifiers' coverage and correctness are central (a weak test suite rewards wrong code).

GRPO

Group Relative Policy Optimization (introduced with DeepSeekMath) removes PPO's value model:

  1. For each prompt, sample a group of G responses (e.g. 8–64).
  2. Score each: r_1 … r_G.
  3. Advantage of every token in response i: A_i = (r_i − mean(r)) / std(r).
  4. Update with the PPO clipped objective plus a KL penalty to the reference.

Scroll sideways to see the whole diagram.

One GRPO update for a promptTrainer → Policy: prompt x: sample G = 8; Policy → Trainer: 8 reasoning traces + answers; Trainer → Verifier: check each answer; Verifier → Trainer: rewards: 1 0 0 1 1 0 0 0; Note: mean 0.375, std 0.48 → A = +1.29 (correct), −0.77 (wrong); Trainer → Reference: token log-probs; Reference → Trainer: for the KL penalty; Trainer: clipped policy-gradient step − β·KL; Trainer → Policy: sync updated weights; [if all 8 rewards are equal]; Note: every advantage is 0: no learning signal, so filter prompts by pass rate.TrainerPolicyrollout engineVerifiertests · answer checkReferencefrozenprompt x: sample G = 88 reasoning traces + answerscheck each answerrewards: 1 0 0 1 1 0 0 0mean 0.375, std 0.48 → A = +1.29 (correct), −0.77 (wrong)token log-probsfor the KL penaltyclipped policy-gradient step − β·KLsync updated weightsIF ALL 8 REWARDS ARE EQUALevery advantage is 0: no learning signal, so filter prompts by pass rate
One GRPO update for a prompt. The policy samples a group of eight responses. A programmatic verifier scores them; three are correct. Each response's advantage is its reward minus the group mean, divided by the group's standard deviation, so correct responses are pushed up and wrong ones down without a learned value model. A KL term to the frozen reference keeps the policy anchored.

The group mean is a baseline that reduces variance without training a critic. Prompts where all responses get the same reward produce zero advantage and no learning signal, so curricula and filtering keep prompts at moderate difficulty (pass rates neither 0 nor 1).

Known issues: dividing by response length and by the group's standard deviation introduces biases (e.g. toward long incorrect answers or toward too-easy or too-hard prompts); variants adjust the normalization. Entropy can collapse, reducing exploration; clipping choices and temperature matter.

Emergent behaviour and length

With outcome rewards only, models learn to write longer reasoning, check their work and backtrack, because those behaviours raise success. Response length often grows over training. Accuracy then scales with test-time compute: more reasoning tokens, or more samples with voting, give better answers on hard problems.

Costs: latency and inference cost grow with thinking length; models can overthink easy questions. Length penalties or budgets trade accuracy for cost.

Reward hacking

Even programmatic rewards can be gamed: special-casing visible tests, exploiting answer-format parsers, or editing test files in agentic tasks. Mitigations: hidden tests, sandbox permissions, robust answer parsing, monitoring transcripts, and penalties for detected exploits.

Distillation

Training a smaller model with SFT on reasoning traces generated by a strong reasoning model transfers much of the capability far more cheaply than running RL on the small model directly.

What to remember

  • RLVR: programmatic rewards in checkable domains.
  • GRPO: group-normalized advantages, no value model.
  • Longer reasoning emerges; manage cost and hacking.