Home / Fundamentals / Evaluation

EvaluationSeniorQuestion 4 of 6

Your new model scores 2 points higher on a 500-item benchmark. Is it better?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask: same items for both models? Greedy or sampled decoding? How was this checkpoint chosen? Framing: "Quantify the noise, run the right paired test, then look at other sources of variance and selection effects before concluding."

5–15 min Core explanation

Standard error at 70% accuracy, n = 500:

SE = √(0.7 × 0.3 / 500) ≈ 0.0205 → ±4.0 points at 95%

Two independent intervals of ±4 points overlap heavily for a 2-point gap. But the models answered the same items, so a paired test is appropriate and more powerful.

McNemar's test:

z ≈ (b − c) / √(b + c)

If the models disagree on 75 items with b = 42, c = 33 (a 1.8-point gap): z = 9 / √75 ≈ 1.04, p ≈ 0.3. Not significant.

import numpy as np
def paired_bootstrap(a, b, iters=10_000, rng=np.random.default_rng(0)):
    a, b = np.asarray(a), np.asarray(b)          # per-item 0/1 correctness
    idx = rng.integers(0, len(a), (iters, len(a)))
    diffs = a[idx].mean(1) - b[idx].mean(1)
    return diffs.mean(), np.percentile(diffs, [2.5, 97.5])

15–25 min Deep dive: other variance and biases

• Decoding randomness: with sampling, run k seeds per item and average; report the spread.
• Training-seed variance: for recipe changes (data mix, hyperparameters), differences between seeds of the same recipe can be ~1–2 points on small benchmarks; compare distributions over seeds.
• Prompt sensitivity: evaluate with several templates; a gain that appears under one template only is fragile.
• Clustered items: many questions per passage or generated from one template are correlated; resample clusters in the bootstrap.
• Selection effects: if the checkpoint or prompt was chosen as the best of many on this same benchmark, the gain is biased upward (winner's curse). Confirm on a held-out set.
• Benchmark validity: label noise (a few percent in many datasets) caps attainable accuracy and adds noise to small differences.

Sample-size planning: detecting a 2-point gain at ~70% with independent samples needs ~8,000 items per arm (80% power); pairing reduces it a lot when models agree on most items, but hundreds of items are rarely enough for 1–2 point claims.

25–32 min What to do instead

1. Pool evidence: several benchmarks measuring the target capability; a consistent direction is more convincing than one significant result.
2. Pre-specify comparisons and thresholds.
3. Report confidence intervals and per-category breakdowns.
4. For product decisions, combine offline gains with online A/B tests.

32–37 min Production example

A team reported a 2.4-point gain from a new data mix on a 600-item benchmark. Re-running both recipes with three training seeds showed seed-to-seed spread of 1.8 points; averaging over seeds gave a 0.9-point gain with a confidence interval spanning zero. On a broader suite of 8 benchmarks the new mix was better on 5 and worse on 3; they kept it only for the domain where it consistently helped.

37–40 min Likely follow-ups

• Is p < 0.05 the right bar? Depends on decision cost; but avoid treating it as a binary truth, and correct for many comparisons.
• Why not a t-test on accuracies? A paired test on per-item outcomes uses the data properly.
• How many judge comparisons for a win rate? Win rate SE is √(w(1−w)/n): at n = 400 and w ≈ 0.5, ±5 points at 95%.

What interviewers listen for

• Computing SE and a paired test on the spot.
• Seeds, prompts, clustering and selection effects.
• A practical decision process.

Red flags

• "Higher is better" with no uncertainty.
• Comparing to a single training run.