Home / Fundamentals / Pre-training & Optimization

Pre-training & OptimizationMidQuestion 1 of 7

Derive the language-modelling loss. How do cross-entropy, perplexity and bits per byte relate?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll derive the training objective from maximum likelihood, compute its gradient, decompose cross-entropy into entropy plus KL, and then explain perplexity and bits per byte, including when each is comparable."

5–15 min Core explanation

An autoregressive model factorizes a sequence's probability with the chain rule:

p_θ(x_1..x_T) = Π_t p_θ(x_t | x_<t)

Maximum likelihood over a dataset maximizes Σ log p_θ(x), i.e. minimizes the average negative log-likelihood per token:

L(θ) = −(1/N) Σ_{tokens} log p_θ(x_t | x_<t)

With logits z ∈ ℝ^V and target y, p = softmax(z):

ℓ = −log p_y = −z_y + log Σ_j e^{z_j}
∂ℓ/∂z_j = p_j − 1[j = y]

That clean gradient (prediction minus one-hot) is why the softmax and cross-entropy are fused in implementations: it's also numerically stable when computed with the log-sum-exp trick.

15–25 min Deep dive

Entropy and KL. For the true next-token distribution p and model q:

H(p, q) = −Σ p log q = H(p) + KL(p ‖ q)

The data's entropy H(p) is a floor no model can beat, which is the irreducible term E in scaling laws. Training reduces only the KL.

Perplexity. PPL = exp(L) with L in nats per token. Intuition: the model is as uncertain as a uniform choice among PPL options. Two caveats:

• It depends on the tokenizer. A tokenizer with longer tokens has fewer, harder predictions per document, so per-token perplexity isn't comparable across vocabularies.
• It's a mean over tokens, so a few very surprising tokens (rare names, numbers) can dominate.

Bits per byte (BPB). Normalize by bytes instead of tokens:

BPB = (Σ NLL in nats) / (ln 2 · total UTF-8 bytes)

Equivalently, BPB = (L / ln 2) · (tokens / bytes). Because bytes don't depend on the tokenizer, BPB compares models with different vocabularies on the same text.

Worked example. A model averages 2.0 nats/token on text with 4.2 bytes per token: PPL = e² ≈ 7.4; BPB = (2.0 / 0.693) / 4.2 ≈ 0.69 bits per byte.

25–32 min Trade-offs and subtleties

• Loss vs. downstream performance: lower loss generally means better models, but small loss differences can hide large differences on specific skills, and instruction-tuned models can have higher perplexity on raw web text while being far more useful.
• What's averaged: padding and prompt tokens must be masked out of the loss; in packed sequences, decide whether document boundaries are masked.
• Label smoothing is rarely used for LLM pre-training; it distorts probabilities that matter for sampling and calibration.
• Calibration: pre-trained models are often well calibrated on next-token probabilities; post-training (RLHF) tends to make them less so.

32–37 min Production example

A team compared two in-house models with different tokenizers using validation perplexity and concluded the new one was worse. Recomputing in bits per byte on the same text reversed the conclusion: the new tokenizer had longer tokens (fewer predictions per document), which inflated per-token perplexity. They switched all reporting to BPB per data domain.

37–40 min Likely follow-ups

• Why not optimize accuracy directly? It's non-differentiable; cross-entropy is a proper scoring rule whose optimum is the true distribution.
• What's the loss of a uniform model? log V nats per token (≈ 10.4 for V = 32k), a useful sanity check at initialization.
• How does temperature relate? Sampling temperature rescales logits at inference; training uses temperature 1.

What interviewers listen for

• The derivation, including the softmax–cross-entropy gradient.
• Entropy plus KL decomposition and the irreducible loss.
• Why perplexity isn't comparable across tokenizers, and how BPB fixes it.

Red flags

• Comparing perplexities across different tokenizers.
• Not knowing the expected initial loss.