Home / Fundamentals / Post-training & Alignment

Reward models, RLHF and DPO

The Bradley–Terry reward model, the KL-regularized RLHF objective and PPO, deriving DPO from the optimal policy, its variants, and online versus offline preference learning.

15 min read

SFT shows the model good responses. Preference optimization teaches it which of two responses is better, which is often easier for people to judge than to write the ideal answer.

Preference data and the reward model

Collect prompts, sample two or more responses, and have annotators (or an AI judge following written principles) pick the better one: pairs (x, y_w, y_l).

The Bradley–Terry model assumes:

P(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) )

Train a reward model r_φ (usually the SFT model with a scalar head) by minimizing −log σ(r_φ(x, y_w) − r_φ(x, y_l)). Only reward differences are identified.

RLHF

Maximize reward while staying close to a reference policy (the SFT model):

max_π  E_{x, y~π} [ r(x, y) ]  −  β · KL( π(·|x) ‖ π_ref(·|x) )

The KL term prevents the policy from drifting into regions where the reward model is wrong (reward hacking) and preserves fluency.

PPO is the classic optimizer: sample responses, score them, estimate per-token advantages with a learned value model (GAE), and update with a clipped ratio objective:

L = E[ min( ρ_t A_t, clip(ρ_t, 1 − ε, 1 + ε) A_t ) ],   ρ_t = π(a_t|s_t) / π_old(a_t|s_t)

It requires four models in memory (policy, reference, reward, value), online generation and careful tuning.

DPO: preference optimization without RL

The KL-regularized objective has a closed-form optimum:

π*(y|x) = π_ref(y|x) · exp( r(x, y) / β ) / Z(x)
⇒ r(x, y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x)

Substitute into Bradley–Terry: the intractable Z(x) cancels in the difference. Parameterizing with the policy being trained gives the DPO loss:

L_DPO = −log σ( β · [ log π_θ(y_w|x)/π_ref(y_w|x) − log π_θ(y_l|x)/π_ref(y_l|x) ] )

It's a classification loss over pairs: no reward model, no sampling, two models in memory (policy and frozen reference). The gradient weights each pair by how wrongly the implicit reward ranks it.

Variants and caveats

  • IPO: a squared loss on the log-ratio margin, avoiding DPO's tendency to push margins without bound on deterministic preferences.
  • KTO: uses unpaired good/bad labels.
  • SimPO / ORPO: reference-free variants with length normalization or an SFT term.
  • Offline vs online: DPO on a fixed dataset learns from responses that a different policy produced; as the policy improves, the data goes stale. Online/iterative DPO regenerates responses from the current policy and relabels them, closing much of the gap with RL.
  • Likelihood displacement: DPO can lower the probability of both chosen and rejected responses while widening the margin; monitor chosen log-probs.

What to remember

  • Bradley–Terry reward modeling; KL-regularized objective.
  • DPO comes from the closed-form optimum; Z cancels.
  • Online data matters; DPO is simpler, RL is more flexible.