Home / Fundamentals / Pre-training & Optimization

Numerical precision and training stability

fp32, fp16, bf16 and fp8 formats, mixed-precision training with master weights and loss scaling, and how to detect and fix loss spikes and divergence at scale.

13 min read

Number formats

Exponent bits set the range; mantissa bits set the precision. bf16 has fp32's range, so values rarely overflow or underflow, but only ~2–3 significant decimal digits.

Mixed-precision training

• Compute matmuls in bf16 (or fp16), which tensor cores execute much faster than fp32.
• Keep a master copy of the weights in fp32. Updates are often far smaller than the weights themselves; added in bf16 they would round to nothing (an update of 1e-4 to a weight of 1.0 is below bf16's resolution).
• Keep optimizer state (Adam's m and v) in fp32.
• Accumulate sums (matmul outputs, softmax normalizers, norms, the loss) in fp32.
• fp16 needs loss scaling: multiply the loss by a large factor so small gradients don't underflow, divide before the update, and adjust the factor dynamically, skipping steps that overflow. bf16 usually doesn't need it.

fp8 training uses fp8 matmuls with scaling factors per tensor or per block, chosen from recent maximum absolute values, while keeping sensitive operations (norms, softmax, optimizer) in higher precision. It can substantially raise throughput on hardware that supports it, at the cost of careful engineering.

Loss spikes and divergence

Large runs sometimes see the loss jump sharply, then either recover or diverge. Common causes:

• Learning rate too high for the current batch size or model size.
• Attention logit growth: query and key norms grow until attention becomes one-hot.
• Output logit drift: the softmax normalizer grows large.
• Bad data batches: long repeated strings, corrupted text, unusual tokens.
• Adam ε and second-moment issues: very small v for rarely updated parameters (e.g. rare-token embeddings) makes their steps huge when a gradient finally arrives.
• Numerical issues: overflow in fp16, or reductions done in low precision.

Monitoring

Track more than the loss:

• global gradient norm and the fraction of clipped steps,
• update-to-parameter norm ratio per layer,
• maximum attention logit per layer,
• activation and residual-stream norms,
• per-domain validation losses.

These usually move before the loss spikes.

The stability playbook

1. Rewind and skip: restart from a checkpoint a few hundred steps before the spike and skip the batches around it. If the spike doesn't recur, the data was likely the trigger.
2. Lower the peak learning rate or lengthen warmup.
3. Architectural fixes: QK-norm for attention logits, z-loss (α · (log Σ e^z)² with a small α such as 1e-4) for output logits, removing bias terms, careful initialization scaling.
4. Optimizer settings: β2 = 0.95, appropriate ε, stricter gradient clipping.
5. Precision: keep softmax, norms and reductions in fp32.
6. Data: find and remove the pathological documents.

Interview angles

• Explain why fp32 master weights are needed even when computing in bf16.
• Compare fp16 and bf16, and when loss scaling is needed.
• Diagnose a loss spike: what you'd look at and what you'd try.
• Explain QK-norm and z-loss.