Estimate latency and throughput for serving a 70B model on 8 H100s.

30-second answer

Weights are 140 GB in bf16, 17.5 GB per GPU with TP = 8. At batch 1, each decode step reads them at 3.35 TB/s: a floor of ~5.2 ms/token, ~8–12 ms in practice with all-reduces and kernel efficiency. At batch 64 with 2k-token contexts, KV reads add ~42 GB (5.2 GB per GPU), giving a ~6.8 ms floor per step for 64 tokens: ~9,000 tokens/s ideal, several thousand in practice, still memory-bound (compute is ~1.1 ms). Prefilling a 2k prompt takes ~2.9e14 FLOPs, ~75 ms at 50% utilization. KV capacity (~450 GB free) bounds the batch.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Assumptions: 70B dense, GQA (8 KV heads, 80 layers, d_head 128), bf16 weights, 8 × H100 SXM 80 GB with NVLink, TP = 8; average prompt 1.5k tokens, output 500 tokens. Framing: "Memory first, then decode at batch 1, then decode at a large batch, then prefill, then what limits throughput."

5–15 min Memory

Weights: 70e9 × 2 B = 140 GB  → 17.5 GB per GPU
Total HBM: 640 GB; reserve ~50 GB for activations, buffers, CUDA graphs
KV space: ~450 GB
KV per token: 2 × 80 × 8 × 128 × 2 B = 320 KB
Average sequence ≈ 2k tokens → 0.66 GB
Max concurrent sequences ≈ 450 / 0.66 ≈ 680

15–25 min Decode and prefill

Batch 1:

Per GPU read per step: 17.5 GB / 3.35 TB/s ≈ 5.2 ms (floor)
+ 2 all-reduces per layer × 80 layers = 160 small all-reduces (~10–20 µs each) ≈ 2–3 ms
Realistic: ~8–12 ms/token → ~80–120 tokens/s for one stream

Batch 64, 2k-token contexts:

KV read: 64 × 2048 × 320 KB ≈ 42 GB total → 5.2 GB per GPU
Per GPU read: 17.5 + 5.2 = 22.7 GB → ≥ 6.8 ms per step
Compute: 2 × 70e9 × 64 ≈ 9e12 FLOPs per step → 1.1e12 per GPU ≈ 1.1 ms at peak

The step is memory-bound (6.8 ms floor vs 1.1 ms of compute):

Throughput ≈ 64 tokens / ~10 ms (realistic) ≈ 6,400 tokens/s per 8 GPUs ≈ 800 per GPU

Batch 256: KV reads ≈ 21 GB per GPU, step floor ≈ 11.5 ms, ~22k tokens/s ideal; per-token latency rises. The batch keeps scaling throughput but with growing TPOT.

Prefill of 1.5k tokens: 2 × 70e9 × 1500 ≈ 2.1e14 FLOPs; at 50% of 8 GPUs' peak (≈ 4e15 FLOP/s) that's ~53 ms.

25–32 min Putting it together

SLO trade-off: if TPOT must be ≤ 30 ms (≈ 33 tokens/s streamed), pick the largest batch that meets it under the measured step-time curve.
Prefill interference: at, say, 10 requests/s × 1.5k tokens = 15k prefill tokens/s ≈ 2.1e15 FLOP/s, about a quarter of the cluster's peak, so chunked prefill or a separate prefill pool is needed.
Cost: at ~$2.5/GPU-hour and ~6,000 tokens/s per node, output cost ≈ $20 / (6,000 × 3,600) per token ≈ $0.9 per million output tokens (before prefill and utilization losses).

Improvements: fp8 weights (8.75 GB per GPU: faster decode), fp8 KV (2× capacity), speculative decoding for low-batch latency, prefix caching, disaggregation; or TP = 4 × 2 replicas if per-GPU memory allows (less all-reduce overhead, more throughput per GPU).

32–37 min Production example

A team estimated ~6k tokens/s per node from first principles and measured 3.9k. Profiling showed CUDA-graph-less decode steps with high CPU launch overhead and small batches due to a conservative KV reservation. Enabling CUDA graphs and raising KV memory utilization reached ~5.8k tokens/s.

37–40 min Likely follow-ups

Why not TP = 8 for an 8B model? Weights fit on one GPU; TP adds communication. Replicate instead.
How does MoE change this? Decode reads only active experts per token at small batch, but at large batch most experts are touched; memory holds all experts.
What about 32k contexts? KV reads dominate; batch drops, and KV quantization or GQA/MLA matters most.

What interviewers listen for

A structured estimate: memory, decode floor, batched decode, prefill.
Correct identification of memory- vs compute-bound.
Translation into SLOs and cost.

Red flags

Using peak FLOPs to estimate decode speed.
Ignoring KV-cache reads at large batch.