Home / Fundamentals / Inference & Serving

The inference roofline: prefill vs decode

Arithmetic intensity, why prefill is compute-bound and decode is memory-bandwidth-bound, how batching moves decode along the roofline, and back-of-envelope latency and throughput estimates.

13 min read

Almost every inference optimization makes sense once you know which resource a phase is limited by. The roofline model answers that.

The roofline

A kernel performs F FLOPs and moves B bytes between HBM and the chip. Its arithmetic intensity is I = F / B. Attainable throughput is:

attainable FLOP/s = min(peak FLOP/s, I × memory bandwidth)

For an H100 SXM: ~989 TFLOP/s dense bf16 and ~3.35 TB/s HBM, so the ridge point is 989 / 3.35 ≈ 295 FLOPs per byte. Below it, a kernel is memory-bound; above it, compute-bound.

Scroll sideways to see the whole diagram.

H100 roofline for LLM inferenceChart: roofline: min(989, 3.35 × I). 1101001,00002505007501,000arithmetic intensity (FLOPs per byte, log scale)attainable TFLOP/sroofline: min(989, 3.35 × I)decode, batch 1: ~3 TFLOP/sdecode, batch 64ridge ≈ 295prefill, 2k tokens
H100 roofline for LLM inference. Attainable throughput is the lower of peak compute (~989 TFLOP/s dense bf16) and bandwidth (3.35 TB/s) times arithmetic intensity; the ridge is at ~295 FLOPs per byte. Decode's weight matmuls have intensity roughly equal to the batch size, so they sit far left on the bandwidth slope; prefill's intensity is roughly the prompt length, so it reaches the compute roof. Weight matmuls only; KV-cache reads keep attention memory-bound.

Decode is memory-bound

Generating one token for one sequence multiplies a single activation vector by every weight matrix: about 2N FLOPs while reading all 2N bytes of bf16 weights. Intensity ≈ 1 FLOP/byte, two orders of magnitude below the ridge.

So single-stream decode speed is set by bandwidth:

time per token ≥ weight bytes / bandwidth
8B model, bf16: 16 GB / 3.35 TB/s ≈ 4.8 ms  → ≤ ~210 tokens/s
70B, bf16, TP = 8: 17.5 GB per GPU / 3.35 TB/s ≈ 5.2 ms (+ all-reduce latency)

Batching is the lever: with b sequences per step, weights are read once and used b times, so intensity ≈ b FLOPs per byte (for the weight matmuls). Throughput grows almost linearly with batch until either the ridge (~300) or memory limits are reached.

But each sequence also reads its own KV cache every step, which batching doesn't amortize. At long contexts, KV reads dominate, and attention during decode stays memory-bound regardless of batch size.

Prefill is compute-bound

The prompt's s tokens are processed in parallel: weights are read once and used s times, so intensity ≈ s. A 2,000-token prompt is well above the ridge.

prefill FLOPs ≈ 2 · N · s  (+ attention)
70B, 2,048 tokens: 2 × 70e9 × 2048 ≈ 2.9e14 FLOPs
on 8 H100s at ~50% utilization: 2.9e14 / (8 × 989e12 × 0.5) ≈ 73 ms

Latency metrics

  • TTFT (time to first token): queueing + prefill. Grows with prompt length.
  • TPOT / ITL (time per output token / inter-token latency): one decode step. Grows with batch size and context length.
  • End-to-end latency ≈ TTFT + output tokens × TPOT. Output length usually dominates.
  • Throughput: tokens per second per GPU, the cost metric. The fundamental trade-off: larger batches raise throughput but also raise TPOT.

What to remember

  • Ridge ≈ 295 FLOPs/byte on H100.
  • Decode intensity ≈ batch size; prefill intensity ≈ prompt length.
  • Single-stream decode speed ≈ bandwidth / weight bytes.