Evaluation

Benchmarks, contamination and what scores mean

What standard benchmarks measure and miss, saturation, prompt and scoring sensitivity, how test data leaks into training, and how to detect and guard against contamination.

LLM-as-judge and preference ratings

Pointwise and pairwise judging, rubrics, known judge biases and how to control them, validating judges against human labels, and Bradley–Terry/Elo ratings for arenas.

The statistics of model comparisons

Standard errors of accuracy, paired tests, how many examples you need, variance from sampling and seeds, the unbiased pass@k estimator, and multiple-comparison traps.

What is pass@k, and how do you estimate it without bias?

How does benchmark contamination happen, and how would you detect and prevent it?

How do you use an LLM as a judge reliably? What biases does it have?

Your new model scores 2 points higher on a 500-item benchmark. Is it better?

Why do LLMs hallucinate, how do you measure it, and how do you reduce it?

Design the evaluation suite for deciding whether to ship a new model version in an LLM product.