Home / Fundamentals / Evaluation

EvaluationSeniorQuestion 3 of 6

How do you use an LLM as a judge reliably? What biases does it have?

30-second answer

Use a strong judge with an explicit rubric or checklist, a reference answer for factual tasks, and pairwise comparison when detecting small differences. Known biases are position (favouring one slot), verbosity, self-preference for its own model family, and rewarding style over correctness; counter them by judging both orders, controlling for length, using a different or multiple judge families, and asking for reasoning before the verdict. Validate on a few hundred human-labelled examples from your domain, comparing judge–human agreement with human–human agreement, and re-validate whenever the judge or prompt changes.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask: what's being judged (chat answers, summaries, agent trajectories), for what decision (regression tracking, model choice, RL reward), and what labels exist. Framing: "Formats, rubric design, biases with mitigations, validation against humans, and operational practices."

5–15 min Core explanation

Scroll sideways to see the whole diagram.

A validated LLM-as-judge pipelineComponents: Eval items (prompts · references); Candidate + baseline (outputs per item); Order swap (A|B and B|A); Judge model (rubric · checklist); Aggregate (win rate · BT ratings · CIs); Versioned rubric (few-shot edge cases); Agreement check (trust or recalibrate); Human labels (200–500 per domain). Flow: Eval items to Candidate + baseline; Candidate + baseline to Order swap; Order swap to Judge model; Judge model to Aggregate; Versioned rubric to Judge model; Judge model to Agreement check (sample); Human labels to Agreement check; Agreement check to Aggregate.Eval itemsprompts · referencesCandidate + baselineoutputs per itemOrder swapA|B and B|AJudge modelrubric · checklistAggregatewin rate · BTratings · CIsVersioned rubricfew-shot edge casesAgreement checktrust or recalibrateHuman labels200–500 per domainsample
A validated LLM-as-judge pipeline. Candidate and baseline outputs are judged in both orders against a versioned rubric; inconsistent verdicts count as ties. A sample is labelled by humans to measure judge agreement, and the judge is trusted only while agreement stays near human–human agreement. Aggregated results report win rates or Bradley–Terry ratings with bootstrap intervals.

Formats:

  • Pointwise rubric: score each output on criteria (accuracy, completeness, tone) with anchored definitions for each score.
  • Checklist: task-specific binary checks ("mentions the 30-day limit", "doesn't promise a refund"); more reliable than holistic scores.
  • Pairwise: "Which response better satisfies the user?" with an option for tie; most sensitive for model comparisons.
  • Reference-guided: include the reference answer or key facts so the judge checks rather than solves.

Prompt design: ask the judge to reason briefly before the verdict; require a structured output; give few-shot examples of borderline cases; separate criteria rather than one overall score.

15–25 min Deep dive: biases

Bias Evidence Mitigation
Position Verdict changes when order swaps Run both orders; inconsistent → tie
Verbosity Longer answers win more Instruction, length-controlled win rates, rubric for concision
Self-preference Judges favour their own family's outputs Judge from another family; panel of judges
Authority and style Confident, formatted answers preferred Reference-guided correctness checks
Capability ceiling Can't verify hard math or code Execution or verifiers instead
Prompt sensitivity Rubric wording shifts scores Fixed, versioned prompts; re-validate on change

25–32 min Validation and operations

  1. Human labels: 200–500 items from your domain, labelled by trained raters with the same rubric; measure inter-rater agreement.
  2. Judge agreement: accuracy and Cohen's κ of judge vs majority human label, compared with human–human agreement; inspect disagreement categories.
  3. Calibrate: adjust the rubric or few-shot examples until agreement is acceptable; for scores, check correlation and per-score confusion.
  4. Monitor: re-validate on judge model upgrades; track judge drift with a fixed anchor set.
  5. Cost: cache judgments; use a cheaper judge for screening and a strong one for decisions.

As an RL reward: a judge that's good enough for evaluation can still be hacked when optimized against; combine with verifiable checks, use a held-out judge for evaluation, and watch for length and style exploitation.

32–37 min Production example

A team tracked answer quality with a 1–10 holistic judge score; it rose steadily while user satisfaction stayed flat. Validation against 300 human labels showed κ = 0.31: the judge rewarded longer, list-heavy answers. They switched to pairwise judgments with position swapping plus a five-item checklist per intent, reaching κ = 0.68 (human–human was 0.72), and the metric began tracking satisfaction.

37–40 min Likely follow-ups

  • How do you aggregate pairwise votes? Win rate against a baseline, or Bradley–Terry ratings with bootstrap confidence intervals.
  • Should the judge see the system prompt? Yes, if it defines success (policies, persona).
  • How do you judge agent trajectories? Outcome checks first (did the task succeed?), then rubric judgments on process (unsafe actions, unnecessary steps).

What interviewers listen for

  • Concrete rubric/checklist design.
  • Named biases with mitigations.
  • Human validation with agreement statistics.

Red flags

  • Using a judge with no validation.
  • One holistic score for everything.