Home / Fundamentals / Evaluation

The statistics of model comparisons

Standard errors of accuracy, paired tests, how many examples you need, variance from sampling and seeds, the unbiased pass@k estimator, and multiple-comparison traps.

11 min read

Most reported improvements are a few points on a few hundred examples, and many are noise.

Standard error of accuracy

For accuracy p on n independent items:

SE = √( p (1 − p) / n )
p = 0.70, n = 500  →  SE ≈ 0.0205  →  95% CI ≈ ±4 points

Comparing two models: pair the data

Both models answer the same items, so compare per-item. Only items where they disagree carry information (McNemar's test):

b = items only model A gets right, c = items only model B gets right
z ≈ (b − c) / √(b + c)

Example: n = 500, models disagree on 75 items, A wins 42 and B 33 (a ~2-point gap): z = 9 / √75 ≈ 1.04, not significant. Pairing is far more powerful than comparing two independent confidence intervals, but a 2-point gain still needs more data or less disagreement.

Sample size: to detect a 2-point difference at ~70% accuracy with independent samples (80% power, α = 0.05) you'd need roughly 2 × (1.96 + 0.84)² × 0.21 / 0.02² ≈ 8,200 items per model; pairing reduces this considerably when models agree on most items.

Other sources of variance

  • Sampling: at temperature > 0, re-run several times or use many samples per item.
  • Training seeds: fine-tuning runs with different seeds can differ by points; compare across seeds for training-recipe decisions.
  • Prompt variance: small prompt changes can move scores; evaluate several prompt templates for robustness.
  • Clustered items: questions from the same document or template aren't independent; use cluster-robust standard errors.

pass@k

pass@k is the probability that at least one of k samples is correct. Estimating it as 1 − (1 − ĉ/n)^k from n samples is biased. The unbiased estimator, with c correct out of n ≥ k samples:

pass@k = 1 − C(n − c, k) / C(n, k)

Scroll sideways to see the whole diagram.

pass@k vs number of samplesChart: p = 0.3; p = 0.1; p = 0.02. 1251020501000255075100samples k (log scale)pass@k (%)p = 0.3p = 0.1p = 0.02p = 0.1, k = 10: 65%
pass@k vs number of samples. pass@k = 1 − (1 − p)^k for problems with per-sample success probability p. Hard problems (small p) that look unsolved at pass@1 become reachable with many samples, which is why pass@k measures coverage rather than single-attempt reliability.

Multiple comparisons

Testing twenty prompt variants and picking the best one at p < 0.05 will find a "significant" winner by chance. Use a held-out confirmation set, correct for multiple comparisons, or pre-register the comparison.

What to remember

  • SE = √(p(1−p)/n): ±4 points at n = 500.
  • Paired tests on disagreements; plan sample sizes.
  • Unbiased pass@k; beware multiple comparisons.