Home / Fundamentals / Inference & Serving

Inference & ServingMidQuestion 7 of 7

Explain temperature, top-k, top-p and min-p sampling, and when to use beam search or constrained decoding.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll define each method precisely with the math, show how they interact, cover beam search and constrained decoding, and then give task-based recommendations, including evaluation implications."

5–15 min Core explanation

The model produces logits z. Then:

def sample(z, T=0.7, top_k=None, top_p=None, min_p=None):
    z = z / max(T, 1e-6)
    p = softmax(z)
    if top_k:  p[p < kth_largest(p, top_k)] = 0
    if top_p:                                   # nucleus
        idx = argsort(p, descending=True); c = cumsum(p[idx])
        keep = c - p[idx] < top_p              # smallest set reaching top_p
        p[idx[~keep]] = 0
    if min_p:  p[p < min_p * p.max()] = 0
    return categorical(p / p.sum())

• Temperature rescales logits: p_i ∝ exp(z_i / T). As T → 0, greedy; T = 1, the model's distribution; T > 1, flatter.
• Top-k: fixed number of candidates; too many when the model is confident, too few when it's uncertain.
• Top-p: the smallest set with cumulative probability ≥ p; adapts to confidence.
• Min-p: relative threshold to the top token; keeps more candidates when the distribution is flat and fewer when peaked, holding up better at high temperature.

15–25 min Deep dive

Why truncate? The tail of a 100k-token vocabulary has many tokens with tiny probabilities each, but together they can hold a noticeable mass. Sampling them occasionally produces incoherent tokens, after which the model conditions on its own error and degrades.

Beam search maximizes sequence log-probability approximately. It works for tasks with one correct output (translation, speech recognition, short structured answers). For open-ended text, high-likelihood sequences are generic and repetitive, which is why sampling is used for chat.

Repetition controls: repetition penalty divides positive logits (and multiplies negative ones) of previously seen tokens; frequency/presence penalties subtract proportional or fixed amounts. Useful for small models; can hurt code or structured outputs that legitimately repeat tokens.

Constrained decoding: at each step, mask tokens that would violate a JSON schema, regular expression or grammar. Efficient implementations precompute which vocabulary tokens are valid in each automaton state. It guarantees syntactic validity, not semantic correctness, and very rigid constraints can reduce answer quality.

Determinism: even at T = 0, batch composition in serving changes floating-point reduction order, so outputs can differ between runs. Fixed seeds and batch-invariant kernels are needed for strict reproducibility.

25–32 min Recommendations

Evaluation implications: report decoding settings with results; greedy and sampled pass@1 differ, and pass@k needs sampling with enough temperature.

32–37 min Production example

A support bot at T = 1.0 occasionally produced off-topic phrases. Lowering to T = 0.5 with top-p 0.9 fixed that but made answers repetitive across users; the team settled on T = 0.7. Separately, their tool-calling path produced invalid JSON in ~1% of calls until they enabled schema-constrained decoding at T = 0.

37–40 min Likely follow-ups

• Does temperature change the argmax? No; it changes everything else.
• Is top-p applied before or after temperature? Typically after: temperature reshapes, then truncation; the order changes results.
• What about reasoning models? Providers often recommend specific settings (e.g. moderate temperature) because greedy decoding can loop in long reasoning.

What interviewers listen for

• Precise definitions and their interaction.
• Beam search vs sampling, and constrained decoding.
• Task-based settings, including evaluation effects.

Red flags

• "Temperature controls creativity" with no mechanism.
• Recommending beam search for chat.