Home / Fundamentals / Evaluation
Benchmarks, contamination and what scores mean
What standard benchmarks measure and miss, saturation, prompt and scoring sensitivity, how test data leaks into training, and how to detect and guard against contamination.
12 min read
A benchmark score is a measurement with a protocol: dataset, prompt format, number of examples (shots), decoding settings, answer extraction and scoring. Change any of these and the number moves.
Families of benchmarks
| Capability | Typical formats | Caveats |
|---|---|---|
| Knowledge | Multiple-choice exams | Saturated; answer-letter extraction quirks |
| Math | Final-answer problems, competition sets | Answer normalization; contamination |
| Code | Unit-tested functions, repository tasks | Test coverage; environment setup |
| Reasoning | Puzzles, science QA | Shortcuts and annotation artefacts |
| Instruction following | Verifiable constraints | Narrow |
| Long context | Retrieval and multi-hop over long inputs | "Needle" tests are easy; synthesis is hard |
| Agents | Multi-step tasks in environments | Variance; environment drift |
| Chat quality | Pairwise human or judge preferences | Style and length biases |
Saturation: once top models score near the ceiling (or the label-noise ceiling), the benchmark no longer separates them; harder successors appear.
Sensitivity: few-shot vs zero-shot, chain-of-thought vs direct answers, answer extraction regexes and even option ordering can move scores by several points. Compare models only under the same harness and settings.
Contamination
Test items (or close paraphrases) appear in training data: web pages that copy benchmarks, solutions on forums, synthetic data generated by models that saw them, or deliberate tuning.
Detection:
- N-gram overlap between test items and the training corpus (e.g. 13-grams), when you have the corpus.
- Membership signals without the corpus: unusually low perplexity on test items, or high probability of rare tokens (min-k% probability methods); the model completing a test item verbatim from a prefix.
- Perturbation gaps: evaluate on freshly written or perturbed versions (new numbers, rephrased questions). A large drop relative to the original suggests memorization.
- Canary strings embedded in benchmark files allow searching for leaks.
Guarding: decontaminate training data against every evaluation you report; keep private held-out sets; use time-split evaluations (problems published after the training cutoff); refresh benchmarks.
What to remember
- A score is meaningless without its protocol.
- Contamination inflates scores; detect with overlap, membership signals and perturbation gaps.
- Private, fresh and time-split sets are the defence.