Distributed Training

Memory and compute arithmetic for training

Where training memory goes (16 bytes per parameter plus activations), activation recomputation, FLOPs per token, MFU, and how to estimate the wall-clock time of a run.

Data parallelism, collectives and ZeRO/FSDP

Distributed data parallel, ring all-reduce and its cost, overlapping communication with the backward pass, and how ZeRO stages and FSDP shard optimizer state, gradients and parameters.

Tensor, pipeline, sequence and expert parallelism

Megatron-style tensor parallelism, pipeline schedules and the bubble, sequence and context parallelism for long inputs, expert parallelism for MoE, and how to map them onto the cluster topology.

Reliability and efficiency at scale

Failures at thousands of GPUs, checkpoint strategy and interval, stragglers and silent data corruption, determinism, and a systematic way to find lost MFU.

Estimate the GPU memory needed to train a 7B-parameter model. Where does it go, and how do you reduce it?

How does data-parallel training synchronize gradients? What does a ring all-reduce cost?

Explain ZeRO stages 1–3 and FSDP. What do they save, and what do they cost?

Compare tensor parallelism and pipeline parallelism. When would you use each?

Plan the pre-training of a 70B dense model on 1.4T tokens with 2,048 H100s. Choose a parallel layout and estimate the time.

Your training run reaches 20% MFU. How do you find and fix the lost throughput?

How do you keep a multi-week training run on thousands of GPUs productive despite failures?