Inference & Serving

The inference roofline: prefill vs decode

Arithmetic intensity, why prefill is compute-bound and decode is memory-bandwidth-bound, how batching moves decode along the roofline, and back-of-envelope latency and throughput estimates.

KV cache, continuous batching and paged attention

How big the KV cache gets, how GQA and quantization shrink it, iteration-level (continuous) batching, paged KV memory, prefix caching, chunked prefill and prefill/decode disaggregation.

Quantization and speculative decoding

Weight-only and weight-activation quantization, outliers and how GPTQ, AWQ and SmoothQuant handle them, KV-cache quantization, and speculative decoding with its acceptance math.

Decoding strategies

Greedy, temperature, top-k, top-p and min-p sampling, beam search, repetition penalties, constrained decoding, and how to choose settings for different tasks.

What is the KV cache? How large does it get, and how do you reduce it?

Explain prefill and decode in roofline terms. Which dominates latency and cost?

Estimate latency and throughput for serving a 70B model on 8 H100s.

How do continuous batching and paged attention improve LLM serving throughput?

How would you quantize an LLM for serving? What are the options and pitfalls?

How does speculative decoding work, why is it lossless, and how much does it speed things up?

Explain temperature, top-k, top-p and min-p sampling, and when to use beam search or constrained decoding.