Explain prefill and decode in roofline terms. Which dominates latency and cost?

30-second answer

Prefill processes all prompt tokens in parallel, reusing each weight across hundreds or thousands of tokens, so its arithmetic intensity is far above the GPU's ridge point (~295 FLOPs/byte on H100) and it's compute-bound; it sets time to first token. Decode produces one token per sequence per step and reads all weights plus each sequence's KV cache, giving intensity roughly equal to the batch size, so it's memory-bandwidth-bound; it sets time per output token. For typical outputs decode dominates end-to-end latency, and batching decode is the main throughput lever.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll introduce the roofline, compute arithmetic intensity for each phase, translate that into TTFT and per-token latency, and then explain what each implies for optimization and scheduling."

5–15 min Core explanation

Roofline: attainable FLOP/s = min(peak, intensity × bandwidth). On an H100 (989 TFLOP/s bf16, 3.35 TB/s), the ridge is ~295 FLOPs/byte.

Scroll sideways to see the whole diagram.

H100 roofline for LLM inference. Attainable throughput is the lower of peak compute (~989 TFLOP/s dense bf16) and bandwidth (3.35 TB/s) times arithmetic intensity; the ridge is at ~295 FLOPs per byte. Decode's weight matmuls have intensity roughly equal to the batch size, so they sit far left on the bandwidth slope; prefill's intensity is roughly the prompt length, so it reaches the compute roof. Weight matmuls only; KV-cache reads keep attention memory-bound.

Prefill: all s prompt tokens go through each weight matrix together. Each bf16 weight (2 bytes) is used in 2s FLOPs, so intensity ≈ s FLOPs/byte. For prompts of a few hundred tokens or more, prefill is compute-bound.

Decode: one new token per sequence per step. Each weight is used for 2b FLOPs with batch b, so intensity ≈ b. For realistic batches (1–256), decode is memory-bound; plus each sequence's KV cache is read every step, with intensity ~1 regardless of batch.

15–25 min Deep dive: numbers

8B model, bf16, one H100:

Decode, batch 1: read 16 GB per token → ≥ 4.8 ms/token (≤ ~210 tok/s)
Decode, batch 32 at 2k context: 16 GB weights + 8.4 GB KV → ≥ 7.3 ms/step, ~4,000 tok/s
Prefill, 2,000 tokens: 2 × 8e9 × 2000 = 3.2e13 FLOPs → ~65 ms at 50% of peak

For a response of 500 tokens at ~15 ms/token in a loaded server: decode ≈ 7.5 s, while prefill of a 2k prompt is tens to hundreds of ms. Decode dominates end-to-end latency for typical outputs; prefill dominates TTFT, and dominates compute for long-input, short-output tasks (classification, RAG summaries).

25–32 min Implications

Goal	Phase	Lever
Lower TTFT	Prefill	Prefix caching, shorter prompts, more FLOPs (fp8), chunking, TP
Lower per-token latency	Decode	Fewer bytes per step (quantization), speculative decoding, smaller batch, TP
Higher throughput	Decode	Bigger batches (continuous batching, paged KV, smaller KV)
Stable latency under load	Both	Chunked prefill, prefill/decode disaggregation

Mixing phases: a long prefill in the same step as ongoing decodes delays all of them (latency spikes). Chunked prefill splits the prompt into chunks interleaved with decodes; disaggregation runs prefill and decode on separate GPU pools, each with the best batch size and parallelism.

Cost asymmetry: API pricing often charges output tokens several times more than input tokens, reflecting this: an input token is processed in a compute-efficient batch, an output token occupies memory-bound decode steps.

32–37 min Production example

A summarization service with 6k-token inputs and 150-token outputs had high TTFT under load, while a chat service had slow token streaming. For summarization, the team enabled prefix caching for shared instructions and chunked prefill, and gave the prefill pool more GPUs. For chat, they used speculative decoding and int8 weights to speed up decode.

37–40 min Likely follow-ups

At what batch size does decode become compute-bound? Around the ridge (~300 on H100) for weight matmuls, but KV reads keep attention memory-bound and memory usually caps batch first.
Why are output tokens priced higher? Memory-bound decode has much lower GPU efficiency per token.
Does TP help decode? Yes, it splits weight reads across GPUs' bandwidth, at the cost of all-reduces.

What interviewers listen for

Arithmetic intensity reasoning with the ridge point.
Numbers for each phase.
Mapping goals to the right phase and lever.

Red flags

"Decode is compute-bound."
Not knowing what TTFT and TPOT are.