Home / Question bank / LLM Fundamentals

LLM FundamentalsJuniorQuestion 10 of 13

What do temperature and top-p do?

30-second answer

Temperature divides the logits before softmax: below 1 sharpens the distribution (more deterministic), above 1 flattens it (more diverse). Top-p samples only from the smallest set of tokens whose cumulative probability exceeds p, cutting off the unreliable tail. For extraction or classification, use low temperature; for brainstorming, higher.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "The model outputs logits over the vocabulary; decoding settings decide how we pick the next token from them. I'll explain temperature, top-k, top-p and related knobs, then give guidance per use case."

5–15 min Core explanation

At each step the model produces a logit z_i for every token. Probabilities come from softmax:

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

Temperature T:

  • T → 0: approaches greedy (always the top token). Deterministic, can loop or be bland.
  • T = 1: the model's learned distribution.
  • T > 1: flatter distribution, more diverse, more errors.

Top-k: keep only the k most likely tokens, renormalize, sample. Fixed k is crude: sometimes the top 3 hold 99% of mass, sometimes 300 plausible options exist.

Top-p (nucleus): keep the smallest set of tokens whose cumulative probability ≥ p (e.g. 0.9). Adapts to how confident the model is.

15–25 min Deep dive: order and other knobs

Typical sampling pipeline per step:

  1. Logits from the model.
  2. Apply penalties (repetition, frequency, presence) and logit bias.
  3. Divide by temperature.
  4. Filter (top-k, then top-p, or min-p).
  5. Softmax and sample.

Other knobs:

  • min-p: keep tokens with probability ≥ min_p × p_max. Scales with confidence; popular in open-source serving.
  • Repetition / frequency / presence penalties: reduce loops and repetitive phrasing.
  • Stop sequences, max tokens: bound output.
  • Seed: helps reproducibility but doesn't guarantee identical outputs on all providers because of batching and floating-point non-determinism.

Some providers recommend adjusting temperature or top-p, not both, since they interact.

25–32 min Guidance by use case

Task Suggested setting
Classification, extraction, SQL, JSON Temperature 0–0.2
Factual Q&A, RAG Low (0–0.4)
General chat Provider default (~0.7–1)
Brainstorming, creative writing Higher (0.9–1.2), top-p ~0.95
Self-consistency voting Moderate temperature, sample several, take majority

Note: some reasoning-focused models fix or limit these parameters; check the provider's documentation.

32–37 min Production example

An extraction pipeline at temperature 0.8 produced 3% invalid JSON and varying field values across retries. Dropping to 0 with structured output mode cut errors to near zero. Separately, the marketing-copy feature kept temperature 1.0 but generated three variants and let users pick, which improved satisfaction more than tuning any single sample.

37–40 min Likely follow-ups

  • Is temperature 0 fully deterministic? Not guaranteed; parallel floating-point arithmetic and batching can change ties and near-ties.
  • What is beam search and why is it rare for chat? It keeps the top-k partial sequences; it favours generic, repetitive text in open-ended generation.
  • Does temperature affect hallucination? Higher temperature increases the chance of sampling low-probability (often wrong) tokens; low temperature doesn't eliminate hallucinations the model is confident about.

What interviewers listen for

  • The softmax formula with T and what each extreme does.
  • Why top-p adapts better than top-k.
  • Practical per-task choices.

Red flags

  • "Temperature controls creativity" with no mechanism.
  • Believing temperature 0 removes hallucinations.