Home / Fundamentals / Evaluation

EvaluationSeniorQuestion 6 of 6

Design the evaluation suite for deciding whether to ship a new model version in an LLM product.

30-second answer

Start from product tasks and failure costs. Build layered evals: deterministic checks (formats, tool calls, unit tests), reference-based task sets sampled from real traffic with labels, rubric or pairwise LLM-judge evals validated against human labels, safety and over-refusal sets, and latency and cost measurements. Version datasets, keep a regression set of past incidents, and stratify by segment. Compare candidate and production on the same items with paired statistics and decision thresholds set in advance, then confirm with shadow traffic and a staged rollout watching online metrics.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask: what the product does, which failures are costly (wrong advice, unsafe actions, format breakage), traffic segments, latency and cost budgets, and who decides. Framing: "Define what good means, build layered evaluations, set decision rules, then connect offline evaluation to online rollout."

5–15 min Requirements

Functional: measure task success per product feature, safety and policy compliance, regressions on known incidents.

Non-functional: results within hours for a candidate; reproducible (versioned datasets, prompts, judges); cheap enough to run on every candidate; trusted (validated against humans).

15–25 min Design

Scroll sideways to see the whole diagram.

Offline evaluation harnessComponents: Golden sets (per feature, safety, tools); Prompts (best per model, frozen); Runner (pinned versions, 3 repeats); Baseline model (current); Candidate model; Graders (exact · programs · judge); Report (feature × category, CIs); Human labels (judge calibration); Gates (pass or block). Flow: Golden sets to Runner; Prompts to Runner; Runner to Baseline model; Runner to Candidate model; Baseline model to Graders; Candidate model to Graders; Graders to Report; Report to Gates; Human labels to Graders (calibrate).Golden setsper feature,safety, toolsPromptsbest per model, frozenRunnerpinned versions,3 repeatsBaseline modelcurrentCandidate modelGradersexact ·programs · judgeReportfeature × category, CIsHuman labelsjudge calibrationGatespass or blockcalibrate
Offline evaluation harness. Both models run on the same golden sets with pinned versions, each with its best (frozen) prompt, repeated to average out sampling noise. Graders range from exact match to calibrated LLM judges; the report shows per-feature, per-category deltas with confidence intervals, and hard gates can block.

Layers, from cheapest to most expensive:

  1. Deterministic checks: JSON/schema validity, tool-call correctness against expected calls, unit tests for generated code, length and latency limits.
  2. Reference-based task sets: items sampled from real traffic (privacy-reviewed), stratified by feature, language and difficulty, labelled with expected answers or key facts.
  3. Judge-based quality: pairwise candidate-vs-production comparisons with position swapping and checklists per intent; judge validated against ~300 human labels per domain.
  4. Safety: harmful-request sets, jailbreak and prompt-injection sets, and over-refusal sets of benign borderline requests.
  5. Regression set: every past incident becomes a permanent test.
  6. Performance: TTFT, tokens/s and cost per task at production settings.

Statistics: paired comparisons on identical items; confidence intervals via bootstrap; pre-set thresholds per metric ("no safety regression beyond CI", "task success non-inferior within 1 point").

25–32 min Deep dive: from offline to online

  • Shadow traffic: run the candidate on mirrored requests; compare outputs with judges and deterministic checks without user impact.
  • Canary / A/B: small traffic share with guardrail metrics (complaints, escalations, thumbs-down, task completion), automatic rollback.
  • Close the loop: sample production failures weekly into the eval sets; track judge drift; retire saturated items.

Trade-offs:

Choice Trade-off
Real-traffic items Representative vs privacy work and labelling cost
LLM judges Scale vs bias; needs validation
Many metrics Coverage vs decision paralysis; define primary metrics
Strict gates Safety vs slower iteration

32–37 min Production example

A team upgraded their model because a public leaderboard showed a big gain. Within a day, tool-call failures rose 3× because the new model formatted arguments differently. Afterwards they built the layered suite above; the next upgrade candidate passed quality evals but failed the over-refusal set (refusing benign medical-billing questions), which they fixed with a system-prompt change before rollout.

37–40 min Likely follow-ups

  • How large should each set be? Enough for the minimum effect you care about: hundreds of items per segment for coarse gates, thousands for 1–2 point decisions.
  • How do you keep evals from being overfit? Hidden test splits, rotating fresh items, separate dev sets for prompt iteration.
  • Who owns the evals? A product-facing team with clear definitions; model teams consume them.

What interviewers listen for

  • Layered evaluations tied to product risk.
  • Paired statistics and pre-set decision rules.
  • The offline-to-online path and feedback loop.

Red flags

  • Relying on public benchmarks for a product decision.
  • No regression or safety sets.