Home / Fundamentals / Inference & Serving

KV cache, continuous batching and paged attention

How big the KV cache gets, how GQA and quantization shrink it, iteration-level (continuous) batching, paged KV memory, prefix caching, chunked prefill and prefill/decode disaggregation.

14 min read

The KV cache

During decoding, keys and values of earlier tokens never change, so they're cached and each step computes projections only for the new token. Size:

KV bytes per token = 2 (K and V) × L × n_kv_heads × d_head × bytes

For a 70B GQA model on 8 × 80 GB GPUs, weights take 140 GB, leaving roughly 450 GB for KV cache after activations and buffers: about 340 concurrent 4k-token sequences, or ~10 at 128k tokens. KV memory determines the maximum batch size, and therefore throughput.

Ways to shrink it: GQA/MQA (fewer KV heads), multi-head latent attention (compressed KV), fp8 or int8 KV cache, sliding-window or hybrid local/global attention layers, and eviction of old tokens (with quality risk).

Continuous batching

Static batching waits until every request in a batch has finished; short requests sit idle while long ones complete, and new requests wait. Continuous (iteration-level) batching schedules at each decode step: finished sequences leave immediately and new ones join. Throughput improvements over static batching are often several-fold under realistic length distributions.

Paged attention

Allocating a contiguous KV region sized for the maximum length wastes most of it (requests rarely reach the maximum) and fragments memory. Paged attention stores the cache in fixed-size blocks (e.g. 16 tokens) with a per-sequence block table, like virtual memory:

• Waste is at most one partly filled block per sequence.
• Blocks can be shared copy-on-write: parallel samples from one prompt, beam search, and common prefixes.
• The attention kernel follows the block table.

Prefix caching

Many requests share a prefix: a system prompt, few-shot examples, a document, earlier turns of a conversation. Caching KV blocks by a hash of the prefix lets a new request skip prefill for the shared part, cutting TTFT and compute. Routing requests with the same prefix to the same replica raises hit rates.

Scheduling prefill and decode

Prefill of a long prompt in the same step as decodes stalls those decodes, causing latency spikes.

• Chunked prefill: split long prompts into chunks (e.g. 512–2,048 tokens) and mix one chunk with ongoing decodes per step, bounding inter-token latency.
• Disaggregation: run prefill and decode on separate GPU pools, transferring the KV cache between them. Each pool can use its own parallelism and batch size and scale independently; the cost is KV transfer bandwidth and complexity.

What to remember

• KV bytes/token = 2·L·n_kv·d_head·bytes; it caps the batch size.
• Continuous batching + paged attention are the baseline of modern servers.
• Prefix caching, chunked prefill and disaggregation tune TTFT vs TPOT.