Home / Fundamentals / Evaluation

EvaluationMidQuestion 1 of 6

What is pass@k, and how do you estimate it without bias?

30-second answer

pass@k is the probability that at least one of k sampled solutions passes the tests, used for code and math where a verifier can pick a correct sample. Plugging the empirical success rate into 1 − (1 − p)^k is biased. Instead draw n ≥ k samples per problem, count c correct, and use 1 − C(n − c, k)/C(n, k), computed stably as a running product, averaged over problems. For n = 20, c = 3, pass@5 ≈ 0.60. Report n, temperature and the k values, use a higher temperature for large k, and note that pass@k overstates single-attempt usefulness.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll define pass@k, show why the naive estimate is biased, derive the unbiased estimator, implement it stably, and discuss how to report and interpret it."

5–15 min Core explanation

For a problem with per-sample success probability p, the probability that at least one of k independent samples is correct is 1 − (1 − p)^k. pass@k for a benchmark is the average over problems.

Naive estimate: sample n solutions, count c correct, plug p̂ = c/n into 1 − (1 − p̂)^k. Because the function is concave in p, Jensen's inequality makes this biased low.

Unbiased estimator: among the n samples, pick k uniformly without replacement. The probability that all k are incorrect is C(n − c, k) / C(n, k). So:

pass@k = E_problems[ 1 − C(n − c, k) / C(n, k) ]

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n - c < k:            # every k-subset contains a correct sample
        return 1.0
    # 1 - prod_{i=n-c+1}^{n} (1 - k/i), avoids huge binomials
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

Example: n = 20, c = 3, k = 5: C(17, 5)/C(20, 5) = 6188/15504 ≈ 0.399, so pass@5 ≈ 0.60. The naive estimate gives 1 − 0.85^5 ≈ 0.56.

15–25 min Deep dive

Scroll sideways to see the whole diagram.

pass@k vs number of samplesChart: p = 0.3; p = 0.1; p = 0.02. 1251020501000255075100samples k (log scale)pass@k (%)p = 0.3p = 0.1p = 0.02p = 0.1, k = 10: 65%
pass@k vs number of samples. pass@k = 1 − (1 − p)^k for problems with per-sample success probability p. Hard problems (small p) that look unsolved at pass@1 become reachable with many samples, which is why pass@k measures coverage rather than single-attempt reliability.
  • Shape: pass@k grows quickly with k when p is moderate; for hard problems with small p, large k reveals capability that pass@1 hides.
  • Variance: use n well above k (e.g. n = 200 for k up to 100) to reduce variance; the estimator needs n ≥ k.
  • Temperature: higher temperatures lower pass@1 but raise pass@k for large k through diversity; report the temperature used, and tune it per k if comparing best-case numbers.
  • What it means: pass@k measures whether correct solutions are reachable by sampling, not single-attempt reliability. It's useful when a verifier (tests) can select among samples, and for RL, where it indicates how much signal sampling can find.
Metric Measures
pass@1 (sampled, averaged over n) Expected single-attempt success
Greedy accuracy Deterministic single attempt
pass@k Coverage with an oracle verifier
maj@k (self-consistency) Accuracy of majority vote over k samples, no verifier
best-of-k with a reward model Practical selection without ground truth

The gap between pass@k and maj@k or best-of-k shows how much a better selector would help.

32–37 min Production example

A team compared two code models: A had pass@1 of 41% vs B's 45%, but A's pass@20 was 78% vs B's 70%. Since their product ran generated code against the user's tests and could try up to 10 candidates, A gave a better user outcome. They reported both numbers with n = 50 samples per problem at temperature 0.8.

37–40 min Likely follow-ups

  • Why is RL said to improve pass@1 but not always pass@k? RL can concentrate probability on solutions the model could already find, raising pass@1 while large-k coverage stays similar.
  • How do you compute confidence intervals? Bootstrap over problems.
  • What if samples aren't independent? Duplicated samples reduce effective diversity; the estimator still holds for drawing from the generated set.

What interviewers listen for

  • The unbiased formula and why the naive one is biased.
  • A stable implementation.
  • Interpretation and reporting conventions.

Red flags

  • Computing pass@k as accuracy times k.
  • Not reporting n and temperature.