Home / Fundamentals / Inference & Serving

Inference & ServingSeniorQuestion 5 of 7

How would you quantize an LLM for serving? What are the options and pitfalls?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask: model size, hardware (fp8 support?), batch sizes and latency targets, quality bar and which tasks matter. Framing: "I'll pick the scheme by bottleneck, explain why LLM quantization is hard, describe the main methods, and finish with an evaluation plan."

5–15 min Core explanation

Quantization maps real values to a small set: q = round(x / s) + z, x̂ = s · (q − z). Choices:

• What: weights only, weights and activations, KV cache.
• Format: int8, int4, fp8 (E4M3), sometimes nf4 or fp4.
• Granularity: per-tensor, per-channel, per-group (e.g. 128 values per scale).
• When: post-training (PTQ, with a small calibration set) or quantization-aware training (QAT).

Pick by bottleneck:

15–25 min Deep dive: why it's hard and how methods cope

Outliers. Some activation channels are 10–100× larger than the rest. With a shared scale, small values round to zero.

• LLM.int8(): keeps outlier channels in fp16, the rest in int8.
• SmoothQuant: divide activations by per-channel factors s_j and multiply the corresponding weight rows by s_j; mathematically equivalent, but moves difficulty from activations (hard) to weights (easy).
• GPTQ: quantizes weights one column at a time and updates remaining columns to compensate, using the inverse Hessian (XᵀX)⁻¹ from calibration activations; int4 with small loss.
• AWQ: finds the small fraction of salient weight channels (those multiplied by large activations) and scales them up before quantization to protect them.
• Rotation methods (e.g. Hadamard transforms): rotate activations and weights so outliers spread across channels, enabling lower-bit activations.

Arithmetic for 70B:

bf16: 140 GB → int8: 70 GB → int4 (group 128, fp16 scales): ~35 + ~1 GB

In int4 the model fits on one 80 GB GPU with ~40 GB for KV cache. Decode at batch 1 approaches 4× fewer bytes per token (2–3× faster in practice due to dequantization work). At large batches, W4A16 kernels are compute-heavier, and fp8 W8A8 often wins.

25–32 min Evaluation plan and pitfalls

• Compare against the bf16 model on your own task suite, not only perplexity: math and code (sensitive to small errors), long-context retrieval, multilingual, tool-call formatting, safety refusals.
• Use enough samples to detect a 1–2 point drop; compare per-example outputs for systematic regressions.
• Check calibration-data mismatch: calibrating on English web text can hurt code or other languages.
• Smaller models degrade more at a given bit width; 4-bit on a 7B model costs more than on a 70B.
• If PTQ loses too much, try QAT or keep sensitive layers (first/last, some attention projections) in higher precision.

32–37 min Production example

A team served a 70B model on 4 × 80 GB GPUs in bf16 (TP = 4). Moving to AWQ int4 on a single GPU per replica cut cost per token by ~2.5× for their low-concurrency chatbot, with no measurable change on their support-QA evals but a 3-point drop on a math-heavy internal benchmark. For the analytics product (math-heavy), they kept fp8 W8A8 on 2 GPUs instead, which preserved accuracy and still improved throughput.

37–40 min Likely follow-ups

• Why not just use int4 activations? Outliers make 4-bit activations very lossy without rotations or QAT.
• What is QLoRA's nf4? A 4-bit format with levels spaced for normally distributed weights, used for fine-tuning a frozen quantized base.
• Does quantization affect speculative decoding? A quantized target can serve as its own draft, or the draft can be quantized aggressively.

What interviewers listen for

• Choosing the scheme from the bottleneck.
• Outliers and at least two methods for handling them.
• A task-specific evaluation plan.

Red flags

• "Int4 is always 4× faster."
• Evaluating only on perplexity.