Home / Question bank / AI System Design
How would you evaluate a new model before switching production traffic to it?
30-second answer
Run it on the offline golden set covering quality, safety and format adherence, and compare against the current model with the same prompts (adjusting prompts if needed). Check latency and cost. Then shadow traffic or a small canary with online metrics and guardrail rates, followed by a staged rollout with a quick rollback path.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
Architecture diagram
Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →
Scroll sideways to see the whole diagram.
0–5 min Clarify and frame
Ask why the switch is happening, because it changes what you optimize for:
- Cost or latency: the bar is "no meaningful quality loss".
- Quality: the bar is "measurably better where it matters, no new failures".
- Deprecation: you must switch; the question is how to minimize regressions.
Also ask which features use the model (chat, extraction, agents, moderation) and which risks matter most (safety, format breakage, tool-use reliability).
Framing: "I'd treat a model change like any risky production deployment: offline evaluation per feature, then shadow traffic and a staged rollout with clear gates and instant rollback."
5–15 min Core explanation: offline evaluation
- Golden sets per feature. Each with real inputs, expected outputs or grading rubrics, and tags by category and difficulty. Include:
- quality cases (accuracy, groundedness, helpfulness),
- format cases (schemas, length limits, Markdown rules),
- safety cases (policy violations, false refusals, jailbreak and injection attempts),
- tool-use cases (right tool, right arguments, recovery from errors).
- Run the candidate with current prompts first, then with adapted prompts. Models differ in verbosity, instruction-following and refusal style; some prompt changes are normal and expected.
- Compare per category, not just the overall score. A model can win on average while losing badly on one category users care about.
- Measure non-quality dimensions: TTFT, tokens/sec, output length distribution, cost per request, error and retry rates.
- Review diffs. Sample cases where old and new outputs differ most or where judges disagree, and have humans look at them.
15–25 min Deep dive: making the comparison trustworthy
- Paired comparisons: same inputs for both models, scored with the same graders. Use pairwise LLM judging with randomized order to reduce position bias.
- Judge independence: if the judge model is from the same family as the candidate, it may favour it. Use a different judge or calibrate against human labels.
- Variance: run important cases several times; report pass rates and confidence intervals rather than single runs.
- Tokenizer differences: context budgets and cost per character change; re-check truncation logic and limits.
- Pinned versions: use explicit model versions, not floating aliases, so what you evaluated is what you deploy.
A simple report template:
| Feature | Metric | Current | Candidate | Δ | Gate |
|---|---|---|---|---|---|
| Support answers | Groundedness | 94% | 95% | +1 | ≥ −1 |
| Extraction | Schema-valid | 99.8% | 99.9% | +0.1 | ≥ 99.5% |
| Safety | False refusals | 2.1% | 4.0% | +1.9 | ≤ 2.5% ✗ |
| All | p95 latency | 3.1s | 2.2s | −0.9s | — |
25–32 min Online rollout and pitfalls
| Stage | What happens | Gate to proceed |
|---|---|---|
| Shadow | Mirror real traffic to the candidate; outputs logged, not shown | Error rates, latency, judge scores on real inputs |
| Canary | 1–5% of users see the candidate | No regression in task success, feedback, guardrail triggers |
| Ramp | 25% → 50% → 100% | Same metrics plus cost tracking |
| Rollback | Feature flag flips back instantly | Pre-agreed trigger thresholds |
Common pitfalls: more verbose answers raising cost and breaking UI layouts; changed refusal behaviour; agents looping because tool-call style changed; subtle formatting differences breaking downstream parsers.
32–37 min Production example
Moving a support bot to a cheaper model passed the overall offline score, but per-category analysis showed a drop on refund-policy questions. Prompt changes fixed most of it. During canary, output length rose 30%, pushing p95 latency up; adding explicit length guidance restored it. Full rollout cut cost significantly with CSAT unchanged, and the old model stayed available behind the flag for two weeks.
37–40 min Likely follow-ups
- How large should eval sets be? Large enough per category to detect the difference you care about; a few hundred overall is a common starting point.
- What about provider-side silent updates? Pin versions, and run a scheduled eval job to detect drift.
- How do you handle multiple features using one model? Switch per feature behind separate flags.
What interviewers listen for
- Per-feature, per-category evals covering quality, format, safety and tools.
- Statistical care and judge bias awareness.
- Staged rollout with gates and rollback.
Red flags
- Switching based on public leaderboard results alone.
- One overall score with no per-category breakdown.