How do you keep a multi-week training run on thousands of GPUs productive despite failures?

30-second answer

Expect failures every few hours at this scale. Checkpoint the full state (weights, optimizer, data position, RNG) sharded and asynchronously, at an interval of about √(2 × checkpoint cost × MTBF), e.g. ~20 minutes. Automate detection, health checks, node replacement from hot spares, and restart from the last checkpoint. Track per-rank timing to find stragglers, watch for silent data corruption via cross-replica checks and replays, and keep runs deterministic so problems can be reproduced. Measure goodput: the fraction of wall-clock time producing useful training progress.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask: cluster size, run length, storage bandwidth, current failure rate, what's automated. Framing: "I'll quantify expected failures, then design checkpointing, detection and recovery, and finally the subtle failures: stragglers and silent data corruption."

5–15 min Core explanation

Failures are routine. Meta reported 419 unexpected interruptions in 54 days of Llama 3 pre-training on 16k GPUs (about one every three hours), roughly 78% hardware-related (GPU, HBM, network, hosts).

The goal is high goodput: the fraction of wall-clock time spent making training progress that isn't lost. Losses come from:

Work since the last checkpoint (lost at each failure).
Time to detect, restart and reload.
Checkpointing overhead itself.

15–25 min Deep dive: checkpointing

Contents: model weights, optimizer state, data-loader position, RNG states, LR scheduler step, gradient-scaler state. For a 70B model: ~1.1 TB.

Making it cheap:

Sharded: each rank writes its own shard in parallel.
Asynchronous: copy to pinned CPU memory (seconds), resume training, write to storage in the background.
Tiered: keep frequent checkpoints in memory or local NVMe (or on peer nodes), and less frequent ones in durable storage.

Interval (Young's approximation):

interval ≈ √(2 · δ · MTBF)
δ = 1 min, MTBF = 3 h = 180 min  →  √360 ≈ 19 min

Expected waste ≈ δ/interval + interval/(2 · MTBF) ≈ 5% + 5% ≈ 10%. Reducing δ (async) and recovery time matters as much as the interval.

25–32 min Detection and recovery

Detection: NCCL timeouts, watchdog for hangs (no step progress), ECC/Xid error monitoring, heartbeat per rank.
Recovery: cordon the bad node, swap in a hot spare, run health checks (GPU burn-in, NCCL all-reduce tests), restart from the latest checkpoint; all automatic.
Elastic options: continue with fewer DP replicas temporarily (adjusting the global batch or accumulation).

Stragglers: a single slow GPU slows everyone. Track per-rank step and collective times; outliers point to thermal throttling, degraded links or bad hosts.

Silent data corruption (SDC): wrong math with no crash. Signs: unexplained loss spikes or NaNs correlated with one host. Mitigations: cross-replica gradient norm checks, periodic deterministic replays on different hardware, and vendor diagnostics.

32–37 min Production example

A 4,096-GPU run had ~70% goodput: checkpoints were synchronous (6 minutes each, every hour), and restarts took ~25 minutes of manual triage. Async sharded checkpoints (20 s of blocking), checkpoints every 20 minutes, automated node replacement from spares and pre-flight health checks raised goodput to ~92%. A recurring loss spike was traced to one host via per-rank gradient norm comparison: SDC on one GPU.

37–40 min Likely follow-ups

How do you resume the data loader exactly? Save sample indices or shard offsets; deterministic shuffling with a stored seed.
What about changing the layout on resume? Store checkpoints in a layout-independent (resharded) format, or convert offline.
Should you checkpoint to the same storage as data? Separate them to avoid contention.

What interviewers listen for

Quantified failure rates and goodput thinking.
Async sharded checkpointing and the interval trade-off.
Automated recovery, stragglers and SDC.

Red flags

"Checkpoint every N hours" with no reasoning.
Treating failures as rare events.