Home / Fundamentals / Distributed Training

Reliability and efficiency at scale

Failures at thousands of GPUs, checkpoint strategy and interval, stragglers and silent data corruption, determinism, and a systematic way to find lost MFU.

11 min read

At thousands of GPUs, hardware failures are routine. Meta reported 419 unexpected interruptions over 54 days of Llama 3 pre-training on 16k GPUs, roughly one every three hours, most from GPU and memory faults.

Checkpointing

  • What to save: weights, optimizer state, data-loader position, RNG states, learning-rate schedule step. For a 70B model: ~140 GB of bf16 weights plus ~840 GB of fp32 master weights and moments, about 1 TB.
  • Sharded and parallel: every rank writes its shard simultaneously; with tens of GB/s of aggregate storage bandwidth, a checkpoint takes tens of seconds.
  • Asynchronous: copy state to CPU memory quickly, resume training, and write to storage in the background.
  • Interval: Young's approximation balances checkpoint cost δ against lost work: interval ≈ √(2 · δ · MTBF). With δ = 1 minute and a failure every 3 hours, checkpoint every ~19 minutes.

Fast recovery

  • Health checks before (re)start: GPU diagnostics, NCCL tests, ECC errors.
  • Hot spares, so a failed node is replaced without waiting for repairs.
  • Automatic restart from the latest checkpoint, with failure classification.
  • Elastic training can continue with fewer data-parallel replicas.

Stragglers and silent errors

  • One slow GPU (thermal throttling, bad link) slows every synchronous step. Track per-rank step times and collective durations to find it.
  • Silent data corruption: a faulty GPU can produce wrong numbers without crashing. Signs: loss spikes or NaNs that move with a specific host. Mitigations: periodic deterministic replay checks and comparing gradient norms across replicas.

Determinism

Fixed data order, seeds and deterministic kernels make it possible to replay a window exactly, which is crucial for diagnosing loss spikes.

Finding lost MFU

Profile one step and account for every millisecond:

  1. Compute efficiency: are matmuls large enough and aligned (dimensions multiple of 64/128)? Fused kernels (flash attention, fused norms)?
  2. Exposed communication: collectives not overlapped with compute; TP spanning nodes; too-small FSDP units or buckets.
  3. Pipeline bubble: too few micro-batches.
  4. Recomputation: full recompute where selective would do.
  5. Data loading and host overhead: CPU-bound tokenization, Python overhead, synchronization points (e.g. .item() calls).
  6. Stragglers.

What to remember

  • Failures every few hours are normal; design for fast, automatic recovery.
  • Checkpoint interval ≈ √(2·δ·MTBF).
  • MFU debugging is an accounting exercise over the step timeline.