Home / Fundamentals / Inference & Serving

Decoding strategies

Greedy, temperature, top-k, top-p and min-p sampling, beam search, repetition penalties, constrained decoding, and how to choose settings for different tasks.

10 min read

The model outputs logits z; decoding turns them into tokens.

Methods

• Greedy: take argmax. Deterministic but prone to repetition loops in open-ended text.
• Temperature: p_i = softmax(z / T)_i. T < 1 sharpens, T > 1 flattens, T → 0 approaches greedy.
• Top-k: keep the k most likely tokens, renormalize. A fixed k is too wide when the model is confident and too narrow when it isn't.
• Top-p (nucleus): keep the smallest set whose cumulative probability is at least p, renormalize. Adapts to the model's confidence.
• Min-p: keep tokens with probability at least min_p × p_max. Scales the cutoff with confidence; behaves well at higher temperatures.
• Beam search: keep the B best partial sequences by total log-probability. Good for short, constrained outputs (translation, ASR); poor for open-ended generation, where it produces bland, repetitive text.
• Penalties: repetition, frequency and presence penalties lower logits of tokens already generated; useful but blunt.

Order matters: implementations typically apply penalties, then temperature, then truncation (top-k/top-p/min-p), then sample.

Constrained decoding

To guarantee valid JSON or a grammar, mask out tokens that would make the output invalid at each step (a finite-state machine or grammar over the tokenizer's vocabulary). It guarantees syntax, not correct content, and overly tight constraints can lower quality because the model can't "think" in its usual format.

Choosing settings

Note that T = 0 isn't perfectly deterministic in serving systems: batching changes floating-point reduction order, which can flip near-ties.

What to remember

• Temperature reshapes; truncation removes the unreliable tail.
• Beam search for constrained tasks, sampling for open-ended ones.
• Constrained decoding for guaranteed formats.