Your large pre-training run has loss spikes. How do you diagnose and fix them?

30-second answer

First check what moved before the spike: gradient norm and clip rate, update-to-weight ratios, max attention logits, output logit magnitudes, activation norms and per-domain losses. Rewind to a checkpoint a few hundred steps earlier and skip the surrounding batches; if it doesn't recur, the data was the trigger. Otherwise lower the peak learning rate or lengthen warmup, add QK-norm for attention logit growth and z-loss for output logits, use β2 = 0.95 and sane ε, keep softmax and reductions in fp32, and remove pathological documents.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask about the setup: model size, precision, optimizer settings, when spikes started, whether loss recovers. Framing: "I'll separate triggers (data, numerics) from underlying fragility (learning rate, logit growth), describe what to monitor, then give a prioritized playbook."

5–15 min Core explanation

A loss spike is a sudden jump that either recovers within some steps or leads to divergence. Common mechanisms:

Mechanism	What happens	Signal
Attention logit growth	q·k magnitudes grow; attention becomes one-hot; gradients become erratic	Max attention logit per layer rising
Output logit drift	Large logits and a large softmax normalizer	Mean log-sum-exp of logits rising
LR too high for current state	Steps overshoot	Gradient norm and update/weight ratios rising, clip rate up
Adam ε / tiny v	Rarely updated parameters get huge steps	Spikes tied to rare tokens
Bad data	Corrupted or repetitive sequences	Spike correlates with specific batches
Numerics	Overflow or low-precision reductions	NaN/Inf, fp16-specific failures

15–25 min Deep dive: diagnosis

Look before the spike. Gradient norms, clip rate, update-to-parameter ratios per layer, max attention logits, output log-sum-exp, residual-stream norms, per-domain losses. Instability usually builds for hundreds of steps.
Rewind and replay. Restart from a checkpoint ~100–500 steps earlier.
Replay the same batches: if the spike repeats, it's deterministic, often data.
Skip the batches around the spike: if training continues smoothly, the data was the trigger, though the model may still be fragile.
Inspect the data in the spike window: extremely long repeated tokens, binary junk, unusual languages, duplicated documents.
Localize in the model: which layers' gradients or logits blew up first.

25–32 min Fixes, in order of cost

Skip the bad data and filter similar documents from the rest of the corpus.
Lower the peak learning rate (or extend warmup); often the single most effective change.
QK-norm to bound attention logits (or logit soft-capping).
z-loss on output logits: 1e-4 · (log Σ e^{z})².
Optimizer: β2 = 0.95, appropriate ε, global-norm clipping at 1.0; consider not decaying embeddings.
Precision: fp32 softmax, norms, losses and reductions; bf16 rather than fp16.
Architecture and initialization: remove bias terms, scale residual output projections, and check parameterization.

Some of these require restarting from scratch (architecture changes); others can be applied on resume (LR, data, clipping).

32–37 min Production example

A 30B run spiked several times after ~40% of training, each time recovering slowly. Max attention logits in three upper layers had been climbing for thousands of steps. Rewinding and skipping batches only postponed the next spike. The team lowered the peak LR by 30% for the remainder of the stable phase and, for the next model, added QK-norm from the start; the next model trained without spikes at the original LR.

37–40 min Likely follow-ups

Should you just let it recover? Some spikes recover, but each wastes compute and can leave lasting damage; repeated spikes indicate fragility.
How does batch size relate? Larger batches reduce noise but often need adjusted LR; increasing batch size can stabilize.
Can determinism help? Deterministic data order and kernels make replays meaningful.

What interviewers listen for

A diagnostic process, not a list of tricks.
Knowledge of logit growth and z-loss/QK-norm.
Rewind-and-skip as a standard operational tool.

Red flags

"Lower the learning rate" with no diagnosis.
Not monitoring anything beyond loss.