Compare RLHF with PPO and DPO. Derive the DPO objective.

30-second answer

RLHF maximizes reward-model score minus β·KL to a reference policy, usually with PPO, which needs online sampling and policy, reference, reward and value models. That objective's optimum is π* = π_ref·exp(r/β)/Z, so r = β log(π*/π_ref) + β log Z; substituting into the Bradley–Terry likelihood cancels Z and gives DPO's loss −log σ(β[log π(y_w)/π_ref(y_w) − log π(y_l)/π_ref(y_l)]), a supervised loss on pairs with only policy and reference models. DPO is simpler and stable but offline; online or iterative DPO and RL with fresh samples generally do better, and RL handles non-preference rewards.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll state the RLHF objective, sketch PPO, derive DPO from the objective's closed-form optimum, then compare them in practice, including online vs offline data."

5–15 min Core explanation: RLHF

Objective, with reward model r, reference policy π_ref (the SFT model) and coefficient β:

max_π  E_{x~D, y~π(·|x)} [ r(x, y) ] − β · KL( π(·|x) ‖ π_ref(·|x) )

PPO in RLHF:

Sample responses from the current policy.
Score with the RM; add a per-token KL penalty −β log(π/π_ref).
Estimate advantages with a value model (GAE).
Several epochs of clipped updates: min(ρA, clip(ρ, 1±ε)A).

Four models (policy, reference, reward, value), online generation, many hyperparameters.

15–25 min Deep dive: deriving DPO

The KL-regularized objective, for each x, is maximized by:

π*(y|x) = (1 / Z(x)) · π_ref(y|x) · exp( r(x, y) / β ),   Z(x) = Σ_y π_ref(y|x) exp(r(x,y)/β)

(It's the Gibbs distribution: the objective equals −β · KL(π ‖ π*) + const.)

Rearrange for the reward:

r(x, y) = β log( π*(y|x) / π_ref(y|x) ) + β log Z(x)

Bradley–Terry depends only on reward differences, so β log Z(x) cancels:

P(y_w ≻ y_l) = σ( β log π*(y_w|x)/π_ref(y_w|x) − β log π*(y_l|x)/π_ref(y_l|x) )

Replace π* with the trainable policy π_θ and maximize the likelihood of the preference data:

L_DPO(θ) = −E[ log σ( β [ log π_θ(y_w|x)/π_ref(y_w|x) − log π_θ(y_l|x)/π_ref(y_l|x) ] ) ]

def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):   # summed log-probs of responses
    margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))
    return -F.logsigmoid(margin).mean()

Gradient: ∝ σ(r̂_l − r̂_w) · [∇ log π(y_w) − ∇ log π(y_l)], where r̂ = β log π/π_ref is the implicit reward. Pairs the model already ranks correctly get little weight.

25–32 min Comparison in practice

	RLHF (PPO)	DPO
Models in memory	4	2
Data	Online samples from current policy	Fixed offline pairs (unless iterative)
Stability	Sensitive to tuning	Stable, supervised-like
Reward signals	Any: RM, rules, verifiers, combinations	Pairwise preferences only
Exploration	Yes	No (offline)
Known issues	Cost, complexity, reward hacking	Overfitting preferences, likelihood displacement, distribution shift

Observations from the literature and practice:

With fixed data, DPO's lack of on-policy samples is its main weakness; online/iterative DPO (sample from the current policy, label with an RM or judge, repeat) recovers much of the gap.
β controls how far the policy moves from the reference, like the RLHF KL coefficient.
Many production pipelines use DPO-style methods for general preference tuning and RL (PPO/GRPO) for verifiable-reward domains.

32–37 min Production example

A team replaced PPO with DPO on 200k offline pairs; training became much cheaper and stable, but gains plateaued and response length grew. Switching to three rounds of iterative DPO (fresh samples from the current model each round, labelled by their reward model, with a length-normalized margin) surpassed the PPO model on human evaluations at a fraction of the cost.

37–40 min Likely follow-ups

Why does DPO lower chosen-response likelihood sometimes? The loss depends only on the margin; both log-probs can fall. An SFT term on chosen responses or variants like IPO help.
What's the reference model for? It anchors the implicit reward; without it, the policy can drift arbitrarily.
When must you use RL? Non-pairwise rewards (tests passing, task success), exploration, multi-step agent tasks.

What interviewers listen for

The full derivation, including why Z cancels.
Practical comparison, especially online vs offline.
Knowledge of DPO failure modes.

Red flags

"DPO doesn't optimize a reward."
Being unable to explain the KL term's role.