How does mixed-precision training work? Compare fp16, bf16 and fp8.

30-second answer

Matmuls run in low precision on tensor cores while an fp32 master copy of weights and the Adam moments are kept, and reductions accumulate in fp32; otherwise small updates round away. fp16 has 5 exponent bits (max 65,504), so gradients underflow without dynamic loss scaling; bf16 keeps fp32's 8-bit exponent with only 7 mantissa bits, so it rarely needs scaling and is the standard. fp8 (E4M3 for weights/activations, E5M2 for gradients) needs per-tensor or per-block scaling factors and higher precision for sensitive ops, but can raise throughput substantially.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll explain how floating-point formats trade range for precision, what mixed-precision training keeps in fp32 and why, fp16 loss scaling, and what fp8 training adds."

5–15 min Core explanation

A float has a sign, exponent (range) and mantissa (precision).

Format	Exp	Mantissa	Max	Relative precision (machine ε)
fp32	8	23	3.4e38	~1.2e-7
fp16	5	10	65,504	~9.8e-4
bf16	8	7	3.4e38	~7.8e-3
fp8 E4M3	4	3	448	~0.125
fp8 E5M2	5	2	57,344	~0.25

Mixed precision:

Matmuls and convolutions in bf16/fp16 on tensor cores, with fp32 accumulation.
fp32 master weights: with bf16's ~3 significant digits, adding an update of 1e-4 to a weight of 1.0 rounds to no change. Keeping fp32 master weights preserves small updates; a bf16 copy is made for the forward pass.
fp32 optimizer state for Adam's m and v.
fp32 for sensitive ops: softmax, normalization statistics, loss, and large reductions.

15–25 min Deep dive

fp16 and loss scaling. fp16's smallest normal value is ~6e-5; many gradients are smaller and underflow to zero. Multiply the loss by a scale factor S (e.g. 2^16) before backward, unscale gradients before the update, and adjust S dynamically: halve it and skip the step on overflow (Inf/NaN), grow it after a run of good steps.

bf16 has fp32's range, so loss scaling is rarely needed. Its low precision is acceptable for activations and gradients when accumulations happen in fp32. It's the standard for LLM training on modern accelerators.

fp8 training.

Two formats: E4M3 (more precision; forward weights and activations) and E5M2 (more range; gradients).
With so little range, values must be scaled: a per-tensor scale from the recent max absolute value (delayed scaling), or finer per-block scales for outliers.
Keep master weights, optimizer state, norms, softmax and the most sensitive layers (often the first and last) in higher precision.
Throughput: fp8 tensor cores offer roughly 2× bf16 peak; end-to-end gains are smaller because not everything runs in fp8.
Risks: outlier activations, divergence late in training; requires careful validation at scale.

Inference quantization is different: post-training weight-only int8/int4 or fp8 is common and doesn't require stable training dynamics.

25–32 min Trade-offs

Choice	Benefit	Risk
fp32 everywhere	Simple, accurate	Slow; 2× memory for activations
fp16 mixed	Fast	Underflow/overflow; needs loss scaling
bf16 mixed	Fast, robust	Low precision; keep sensitive ops in fp32
fp8 mixed	Fastest matmuls	Scaling complexity, outliers, stability

Memory: activations stored in bf16 halve activation memory versus fp32, which often matters more than weight memory during training.

32–37 min Production example

A team moved a 13B model's training from bf16 to fp8 matmuls with per-tensor delayed scaling. Throughput rose ~30%, but loss diverged at ~60% of training. Switching to finer-grained (per-block) scaling for activations, and keeping the attention output projection and LM head in bf16, matched the bf16 loss curve within noise while keeping most of the speedup.

37–40 min Likely follow-ups

Why accumulate in fp32? Summing many low-precision products loses small contributions; fp32 accumulators avoid systematic error.
What is stochastic rounding? Rounding up or down with probability proportional to the remainder, so small updates survive in expectation; an alternative to master weights in some settings.
Do gradients need fp32 all-reduce? Often reduced in bf16 for bandwidth, with care; some setups keep fp32 for accuracy.

What interviewers listen for

Range vs. precision, with the format table.
Why master weights and fp32 accumulation are needed.
fp16 loss scaling mechanics, and fp8 scaling granularity.

Red flags

"bf16 is more precise than fp16."
Not knowing why small updates vanish in low precision.