Your training run reaches 20% MFU. How do you find and fix the lost throughput?

30-second answer

MFU is useful model FLOPs (≈6N per token plus attention) per second divided by peak; 20% for a dense model on H100s leaves a lot on the table. Profile a step and account for time: exposed communication (TP across nodes, non-overlapped gradient reduce, small FSDP units), pipeline bubble (too few micro-batches), full recomputation where selective suffices, unfused or misaligned kernels, data-loader or Python host overhead and synchronization points, and stragglers visible in per-rank step times. Fix the biggest bucket first, re-measure, and repeat.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask: model size and architecture, parallel layout, sequence length, micro-batch size, hardware, how MFU is computed. Framing: "I'll validate the measurement, then profile one step and attribute lost time to categories, fixing the largest first."

5–15 min Core explanation

MFU = (model FLOPs per token × tokens per second) / (GPUs × peak FLOP/s)
model FLOPs per token ≈ 6N + 12 · L · h · s   (training)

Recomputation FLOPs aren't counted (that's hardware FLOPs utilization, HFU).

Example: 7B model on 64 H100s at 4,500 tokens/s per GPU:

6 × 7e9 × 4500 ≈ 1.9e14 FLOP/s per GPU  →  19% of 989e12

A well-tuned run of this size can reach ~40% (~9,400 tokens/s per GPU).

First, check the measurement: correct peak (dense bf16, not sparsity figures), attention FLOPs included consistently, and time measured over steady-state steps (not including evaluation or checkpointing).

15–25 min Deep dive: attribute the step time

Profile one step (PyTorch profiler / Nsight) and build a breakdown:

Category	Signs	Fixes
Kernel efficiency	Matmuls far from peak; many small kernels	Larger micro-batch, dimensions multiples of 128, fused kernels (flash attention, fused norm/RoPE/loss), `torch.compile`
Exposed communication	Collectives on the timeline with GPUs idle	Overlap (bucketing, prefetch), TP within node only, larger FSDP units, HSDP
Pipeline bubble	Idle stages at start/end of step	More micro-batches, interleaving, balance stages
Recomputation	HFU ≫ MFU	Selective rather than full recompute where memory allows
Host overhead	GPU gaps between kernels	CUDA graphs, remove `.item()` / sync points, faster data loader
Data loading	Step starts waiting for input	Pre-tokenized data, more workers, prefetch
Stragglers	Some ranks consistently slower	Per-rank timing, replace bad nodes

25–32 min Trade-offs

Larger micro-batches improve matmul efficiency but need more memory, so they may require more recomputation; the net effect must be measured.
More model parallelism fits larger batches but adds communication.
Fusions and compilation can complicate debugging and numerics checks.
Lower precision (fp8) raises peak throughput, but MFU should be measured against the same peak you claim.

32–37 min Production example

A 7B run at 19% MFU. Breakdown: 30% of step time was exposed FSDP all-gathers across nodes (full sharding over 64 GPUs), 12% host gaps from a logging call that synchronized every step, and the micro-batch was 1 at 4k tokens. Switching to HSDP (shard within node), removing the sync, and raising micro-batch to 2 with selective recompute reached 38% MFU, doubling throughput.

37–40 min Likely follow-ups

Why not 100% MFU? Non-matmul ops (norms, softmax, elementwise), communication, memory-bound phases and imperfect kernels.
How does sequence length affect MFU? Longer sequences increase attention FLOPs (counted) and can improve efficiency, until memory forces smaller batches.
What's goodput? Effective progress per wall-clock time including failures and restarts, which is what ultimately matters.

What interviewers listen for

A correct MFU definition with a worked number.
Systematic attribution using a profile.
Concrete fixes per category.

Red flags

Guessing fixes without profiling.
Confusing MFU and HFU.