Home / Fundamentals / Evaluation

LLM-as-judge and preference ratings

Pointwise and pairwise judging, rubrics, known judge biases and how to control them, validating judges against human labels, and Bradley–Terry/Elo ratings for arenas.

12 min read

Open-ended outputs (chat answers, summaries, agent transcripts) have no single reference answer. A strong model, prompted with a rubric, can grade them at scale.

Scroll sideways to see the whole diagram.

A validated LLM-as-judge pipelineComponents: Eval items (prompts · references); Candidate + baseline (outputs per item); Order swap (A|B and B|A); Judge model (rubric · checklist); Aggregate (win rate · BT ratings · CIs); Versioned rubric (few-shot edge cases); Agreement check (trust or recalibrate); Human labels (200–500 per domain). Flow: Eval items to Candidate + baseline; Candidate + baseline to Order swap; Order swap to Judge model; Judge model to Aggregate; Versioned rubric to Judge model; Judge model to Agreement check (sample); Human labels to Agreement check; Agreement check to Aggregate.Eval itemsprompts · referencesCandidate + baselineoutputs per itemOrder swapA|B and B|AJudge modelrubric · checklistAggregatewin rate · BTratings · CIsVersioned rubricfew-shot edge casesAgreement checktrust or recalibrateHuman labels200–500 per domainsample
A validated LLM-as-judge pipeline. Candidate and baseline outputs are judged in both orders against a versioned rubric; inconsistent verdicts count as ties. A sample is labelled by humans to measure judge agreement, and the judge is trusted only while agreement stays near human–human agreement. Aggregated results report win rates or Bradley–Terry ratings with bootstrap intervals.

Formats

  • Pointwise: score one output against a rubric (e.g. 1–5 per criterion, or pass/fail per checklist item). Stable across time; good for regression tracking.
  • Pairwise: which of two outputs is better? More sensitive to small differences and closer to how humans compare; results are relative.
  • Reference-guided: give the judge a reference answer or key facts; more reliable for factual tasks.
  • Checklists: decompose quality into binary questions ("cites the policy section", "states the refund amount"); easier to validate.

Known biases

Bias Description Mitigation
Position Prefers the first (or second) answer Judge both orders; count inconsistent pairs as ties
Verbosity Prefers longer answers Instruct against it; length-controlled analysis
Self-preference Prefers outputs from its own model family Use a different judge family or several judges
Style over substance Rewards formatting and confident tone Rubrics on correctness; reference answers
Limited ability Can't grade problems it can't solve Reference answers; verifiers instead

Validate the judge

Treat the judge as a classifier: collect a few hundred human labels on your task, and measure agreement (accuracy and Cohen's κ) between judge and humans, compared with human–human agreement. Strong judges on general chat agree with humans about as often as humans agree with each other (around 80% on pairwise preferences in published studies), but agreement on your domain must be measured. Re-validate when the judge model or prompt changes.

Ratings from pairwise comparisons

Arena-style evaluations collect many pairwise votes and fit a Bradley–Terry model: P(i beats j) = σ(θ_i − θ_j), often reported on an Elo scale where a 100-point gap means ~64% expected win rate. Report confidence intervals (bootstrap); control for style (length, formatting) when possible, because voters reward it.

What to remember

  • Pairwise for sensitivity, pointwise/checklists for tracking.
  • Swap positions, control length, validate against humans.
  • Bradley–Terry ratings with confidence intervals.