Home / Fundamentals / Inference & Serving

Quantization and speculative decoding

Weight-only and weight-activation quantization, outliers and how GPTQ, AWQ and SmoothQuant handle them, KV-cache quantization, and speculative decoding with its acceptance math.

13 min read

Both techniques attack the same bottleneck: memory-bound decode.

Quantization

Scheme What's quantized Main benefit Typical quality cost
W8A16 / W4A16 (weight-only) Weights Fewer bytes per decode step; fits on fewer GPUs int8 ≈ none; int4 small, larger for small models
W8A8 (int8 or fp8) Weights and activations Faster matmuls on int8/fp8 tensor cores; helps prefill and large batches Small with good calibration
KV cache int8/fp8 Keys and values 2× more sequences or longer contexts Usually small

Why it's hard: LLM activations have a few channels with very large values (outliers). Quantizing them with a shared scale crushes everything else.

  • GPTQ: quantizes weights column by column, using second-order (Hessian) information from calibration data to compensate the error in the remaining columns.
  • AWQ: protects the weight channels that multiply large activations by scaling them before quantization.
  • SmoothQuant: moves activation outliers into weights with a per-channel scale, making W8A8 feasible.
  • Granularity: per-channel or per-group (e.g. 128 weights per scale) scales trade a little memory for accuracy.

Arithmetic: a 70B model in int4 is ~35 GB (+ ~1 GB of group scales), fitting on one 80 GB GPU with room for KV cache, versus 140 GB in bf16. Single-stream decode can approach the 4× bandwidth reduction minus dequantization overhead (~2–3× in practice). At large batch sizes, weight-only quantization helps less because decode becomes more compute-bound; fp8 W8A8 is then better.

Always evaluate on your tasks: aggregate benchmarks hide regressions in math, code, long context and multilingual text.

Speculative decoding

A cheap draft proposes k tokens; the target model scores all of them in one forward pass (cheap, because decode is memory-bound and processing k+1 tokens costs almost the same as 1).

Scroll sideways to see the whole diagram.

One round of speculative decodingEngine → Draft model: context …“The cat”; Draft model: draft 4 tokens: sat · on · a · mat; Draft model → Engine: tokens + draft probs q; Engine → Target model: one forward pass over context + 4 drafts; Target model → Engine: target probs p at 5 positions; Note: accept with prob min(1, p/q): sat ✓ on ✓ a ✓ mat ✗; Engine: sample correction from norm(max(0, p − q)): “the”; Note: 4 tokens emitted for one target pass; next round starts from “… sat on a the”.EngineDraft model~1B or extra headsTarget model70Bcontext …“The cat”draft 4 tokens: sat · on · a · mattokens + draft probs qone forward pass over context + 4 draftstarget probs p at 5 positionsaccept with prob min(1, p/q): sat ✓ on ✓ a ✓ mat ✗sample correction from norm(max(0, p − q)): “the”4 tokens emitted for one target pass; next round starts from “… sat on a the”
One round of speculative decoding. The draft proposes four tokens autoregressively. The target scores all of them in a single forward pass, which costs about one decode step because decode is memory-bound. Tokens are accepted left to right with probability min(1, p/q); at the first rejection a correction is sampled from the residual distribution, so the output distribution exactly matches the target's.

Acceptance rule (preserves the target distribution exactly): for each drafted token x with draft probability q(x) and target probability p(x), accept with probability min(1, p(x)/q(x)). On the first rejection, sample a replacement from normalize(max(0, p − q)) and discard the rest. If all are accepted, sample one bonus token from the target.

Expected tokens per target pass with per-token acceptance rate α and k drafts:

E[tokens] = (1 − α^(k+1)) / (1 − α)
α = 0.8, k = 4  →  (1 − 0.328) / 0.2 ≈ 3.4

If drafting 4 tokens costs ~20% of a target step, speedup ≈ 3.4 / 1.2 ≈ 2.8×; real systems commonly see 2–3×.

Draft options: a small model from the same family (same tokenizer), extra decoding heads on the target (Medusa), feature-level drafters (EAGLE), n-gram/prompt lookup (great when output copies input, e.g. code edits or RAG).

When it helps less: at large batch sizes the target step is closer to compute-bound, so verifying extra tokens isn't free; with high temperature, acceptance drops.

What to remember

  • Weight-only int4 for memory-bound small-batch serving; fp8 W8A8 for throughput.
  • Outliers are the core difficulty.
  • Speculative decoding is lossless; expected tokens per pass = (1 − α^{k+1})/(1 − α).