Home / Fundamentals / Distributed Training

Distributed TrainingMidQuestion 2 of 7

How does data-parallel training synchronize gradients? What does a ring all-reduce cost?

30-second answer

Each replica computes gradients on its slice of the batch, then gradients are summed with an all-reduce so every replica applies the same update. A ring all-reduce is a reduce-scatter followed by an all-gather, each N−1 steps of S/N bytes, so each GPU sends 2(N−1)/N·S bytes, nearly independent of N. For 14 GB of bf16 gradients on 8 GPUs that's ~24.5 GB per GPU, ~55 ms on NVLink. Gradients are bucketed and reduced during the backward pass so communication overlaps compute; at large scale hierarchical and tree algorithms reduce latency.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll describe the data-parallel step, explain all-reduce algorithms with their cost, show a concrete number, then cover overlap, bucketing and what changes at very large scale."

5–15 min Core explanation

One data-parallel step:

  1. Each of N replicas receives a different micro-batch.
  2. Forward and backward produce local gradients.
  3. All-reduce sums gradients (divide by N for the mean).
  4. Each replica applies the identical optimizer update, so replicas stay in sync.

This is equivalent to training on the global batch on a single device (up to floating-point summation order).

Naive approach: gather all gradients to one parameter server and broadcast back: the server's link carries N · S bytes, a bottleneck.

Ring all-reduce: arrange GPUs in a ring and split gradients into N chunks.

Scroll sideways to see the whole diagram.

Ring all-reduce on 4 GPUs[reduce-scatter · 3 steps, all GPUs send in parallel]; GPU 0 → GPU 1: step 1: chunk a; GPU 1 → GPU 2: chunk b; GPU 2 → GPU 3: chunk c; GPU 3 → GPU 0: chunk d; Note: each receiver adds its own copy; steps 2–3 pass the partial sums on; GPU 0 → GPU 1: steps 2–3: partial sums d, c; Note: each GPU now owns one fully summed chunk (GPU 0: b, GPU 1: c, GPU 2: d, GPU 3: a); [all-gather · 3 steps]; GPU 0 → GPU 1: summed chunk; GPU 1 → GPU 2: summed chunk; GPU 2 → GPU 3: summed chunk; GPU 3 → GPU 0: summed chunk; Note: after 3 steps every GPU holds all 4 summed chunks.GPU 0chunks a b c dGPU 1GPU 2GPU 3REDUCE-SCATTER · 3 STEPS, ALL GPUS SEND IN PARALLELstep 1: chunk achunk bchunk cchunk deach receiver adds its own copy; steps 2–3 pass the partial sums onsteps 2–3: partial sums d, ceach GPU now owns one fully summed chunk (GPU 0: b, GPU 1: c, GPU 2: d, GPU 3: a)ALL-GATHER · 3 STEPSsummed chunksummed chunksummed chunksummed chunkafter 3 steps every GPU holds all 4 summed chunks
Ring all-reduce on 4 GPUs. Gradients are split into 4 chunks. In the reduce-scatter phase each GPU passes a chunk to its neighbour, which adds its own copy; after 3 steps each GPU owns the full sum of one chunk. The all-gather phase circulates the summed chunks for 3 more steps. Each GPU sends 2 × 3/4 of the gradient size in total.
  • Reduce-scatter (N − 1 steps): each GPU sends one chunk to its neighbour, which adds it to its own; after N − 1 steps, each GPU holds the full sum for one chunk.
  • All-gather (N − 1 steps): circulate the summed chunks until every GPU has all of them.

Each step moves S/N bytes per GPU, so per GPU:

bytes sent = 2 · (N − 1) · S / N  ≈ 2S

15–25 min Deep dive: numbers and overlap

7B model, bf16 gradients, S = 14 GB, N = 8 GPUs in one node:

2 × 7/8 × 14 GB = 24.5 GB per GPU
NVLink ~450 GB/s per direction  →  ~55 ms

Across nodes at ~50 GB/s per GPU (400 Gb/s NIC each): ~0.5 s. A backward pass for a large batch takes longer than that, so with overlap most of it can be hidden.

Overlap with backward. Gradients for the last layers are ready first. Frameworks group gradients into buckets (e.g. 25 MB); when a bucket is complete, its all-reduce starts asynchronously while backward continues on earlier layers. Bucket size trades latency (too small: many small messages) against overlap (too large: waits too long to start).

Latency term. Ring all-reduce has 2(N − 1) sequential steps, each with a fixed latency α: total ≈ 2(N−1)·α + 2·(N−1)/N · S/β. At thousands of GPUs, the latency term matters, so libraries use:

  • Hierarchical: reduce within nodes over NVLink, all-reduce across nodes, broadcast within nodes.
  • Tree algorithms: O(log N) latency.
  • In-network reduction on some switches.

25–32 min Trade-offs and practicalities

  • Global batch size grows with N. Beyond the critical batch size, more data parallelism stops helping per-token efficiency; increase model parallelism instead or accept fewer replicas.
  • Gradient compression (fp16/bf16 reduce, low-rank or 1-bit methods) saves bandwidth with some accuracy risk.
  • Determinism: summation order changes results slightly; deterministic algorithms trade speed.
  • Gradient accumulation reduces how often you synchronize.

32–37 min Production example

A team moved fine-tuning from one 8-GPU node to four nodes and saw only 2.3× speedup. Profiling showed all-reduce exposed after backward: their small model's backward was too short to hide inter-node communication, and the bucket size was tiny. Larger buckets, gradient accumulation of 2 micro-batches, and hierarchical all-reduce brought scaling to 3.6×.

37–40 min Likely follow-ups

  • How do you all-reduce in ZeRO-2? Reduce-scatter only: each rank keeps the reduced shard it owns.
  • Why is ring all-reduce "bandwidth optimal"? Every GPU must send and receive at least ~2(N−1)/N · S bytes to produce the full sum everywhere.
  • What about stragglers? Synchronous all-reduce waits for the slowest rank.

What interviewers listen for

  • Reduce-scatter + all-gather decomposition and the 2(N−1)/N · S cost.
  • Overlap via bucketing.
  • Awareness of latency at scale and hierarchical algorithms.

Red flags

  • Parameter-server reasoning without noticing the bottleneck.
  • Believing all-reduce cost grows linearly with N per GPU.