Compare tensor parallelism and pipeline parallelism. When would you use each?

30-second answer

Tensor parallelism splits each layer's matrices across GPUs (column-parallel then row-parallel in the MLP, by heads in attention), needing two all-reduces of b·s·h activations per layer per direction, so it's kept within a node on NVLink. Pipeline parallelism splits layers into stages and sends only boundary activations point-to-point, so it works across nodes, but idles for (p−1)/(m+p−1) of the time unless there are many micro-batches; 1F1B bounds activation memory and interleaving shrinks the bubble. Typical layouts use TP = 8 within a node, a small PP degree across nodes, and data parallelism for the rest.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll explain how each splits the model, derive their communication and idle-time costs, and then explain how they're combined with data parallelism according to the hardware topology."

5–15 min Core explanation: tensor parallelism

Split matrices within each layer across t GPUs (Megatron-LM):

MLP:  Y = GeLU(X · A) · B
A split by columns: each GPU computes GeLU(X · A_i) independently (no comm)
B split by rows:    each GPU computes GeLU(X · A_i) · B_i, a partial sum
all-reduce:         Y = Σ_i GeLU(X · A_i) · B_i

Attention splits by heads: each GPU computes its heads, then the output projection is row-parallel with one all-reduce.

Communication: 2 all-reduces per layer forward, 2 backward, each of b · s · h activations.
Benefit: weights, gradients, optimizer state and most activations are divided by t; each GPU does 1/t of the FLOPs per layer.
Cost: very frequent, latency-sensitive communication on the critical path. Kept within a node on NVLink, typically t ≤ 8.
Sequence parallelism adds a split along the sequence for norms and dropout, cutting the remaining replicated activations by t.

15–25 min Deep dive: pipeline parallelism

Split the layers into p contiguous stages on different GPUs. Stage i sends activations to stage i+1 (forward) and gradients back (backward), point-to-point.

Scroll sideways to see the whole diagram.

Pipeline bubble vs micro-batches. Fraction of time each GPU idles in a GPipe or 1F1B schedule, (p − 1) / (m + p − 1), for p pipeline stages and m micro-batches per step. Deeper pipelines need many more micro-batches to keep the bubble small; interleaving divides it by roughly the number of chunks per GPU.

Without micro-batching only one stage works at a time. With m micro-batches:

bubble = (p − 1) / (m + p − 1)
p = 4, m = 16  → 3/19 ≈ 16%
p = 8, m = 32  → 7/39 ≈ 18%
p = 8, m = 64  → 7/71 ≈ 10%

Schedules:

GPipe: all forwards then all backwards; stores activations of all m micro-batches.
1F1B: alternates forward and backward after a warmup; stores at most p micro-batches' activations; same bubble.
Interleaved 1F1B: v chunks per GPU; bubble ≈ (p − 1)/(v · m + p − 1), at the cost of v× more point-to-point messages.
Zero-bubble variants split backward into input-gradient and weight-gradient computation to fill idle slots.

Other PP costs: stage imbalance (embedding and LM head layers make first and last stages heavier), and the requirement m ≥ several × p, which ties the global batch size to the pipeline depth.

25–32 min Comparison and combination

	Tensor parallel	Pipeline parallel
Splits	Each layer's matrices	Layers into stages
Communication	All-reduce of activations, 4× per layer (fwd + bwd)	Point-to-point at stage boundaries
Bandwidth need	Very high	Low
Idle time	None (but comm on critical path)	Bubble (p−1)/(m+p−1)
Best placement	Within node (NVLink)	Across nodes
Constraint	Matmuls get small at high t	Needs many micro-batches

Combining (3D parallelism): TP within the node, PP across a few nodes, DP (with ZeRO-1) across the rest: GPUs = t × p × d. Choose the smallest t × p that fits memory, because both reduce efficiency. Increase t first up to 8 if per-layer activations are the problem; add p when the model's state still doesn't fit.

32–37 min Production example

A team scaled a 175B model to 1,024 GPUs with TP = 16 spanning two nodes; inter-node all-reduces made MFU ~22%. Re-laying out as TP = 8 within each node and PP = 16 (with interleaving, v = 2, and m = 64 micro-batches) raised MFU to ~40%.

37–40 min Likely follow-ups

How does FSDP compare? FSDP shards state but each GPU computes full layers; good up to tens of billions of parameters, combined with TP beyond.
Where does context parallelism fit? For very long sequences, splitting the sequence across GPUs within or across nodes.
How do you balance stages? Assign fewer transformer blocks to the stages with the embedding and LM head.

What interviewers listen for

Column/row split and the all-reduce count.
The bubble formula with numbers.
Topology reasoning for placement.

Red flags

Putting tensor parallelism across slow inter-node links.
Not knowing what the pipeline bubble is.