A framework for AI system design interviews

A repeatable structure for 45-minute LLM system design rounds, with what interviewers listen for at each step.

10 min read

AI system design rounds look like classic system design, plus questions about model choice, evaluation, safety and cost. Use a structure so you cover everything without rambling.

1. Clarify the problem (5 min)

Users and use case. Who asks what? Example: support agents asking about internal policies.
Quality bar. What does a wrong answer cost? A typo in a marketing draft is cheap; a wrong medication dose is not.
Scale. Requests per second, peak vs. average, corpus size, number of tenants.
Latency target. Interactive chat (time-to-first-token under about 1 second, streaming) vs. batch.
Constraints. Data residency, privacy, budget, on-prem requirements.

2. Define success metrics (3 min)

State them out loud. Interviewers want to see that you think about measurement before architecture.

Product: task success rate, deflection rate, user satisfaction.
Model quality: groundedness, accuracy on a golden set, refusal rate on in-scope questions.
System: p50/p95 latency, time-to-first-token, availability, cost per request.
Safety: policy-violation rate, prompt-injection success rate in red-team tests.

3. High-level architecture (10 min)

A typical shape:

Client → API gateway (auth, rate limits)
       → Orchestrator
            ├─ Input guardrails (PII, injection, policy)
            ├─ Retrieval (hybrid search + rerank)   ← Index ← Ingestion pipeline
            ├─ Model router → LLM provider(s) / self-hosted model
            ├─ Tools (APIs, DB queries) with permission checks
            └─ Output guardrails (policy, grounding, format)
       → Stream response to client
Logging / tracing → Eval pipeline, dashboards, feedback store

4. Deep dives (20 min)

Pick two or three areas the interviewer cares about. Common ones:

Model selection and routing: small fast model for classification and easy queries, larger model for hard ones; fallback across providers.
Retrieval: chunking, hybrid search, freshness, permissions.
Latency: streaming, caching, parallel tool calls, shorter outputs.
Cost: token budgets, prompt caching, smaller models, batching.
Safety: guardrails, prompt-injection defence, human review.
Evaluation: offline golden sets, online A/B tests, monitoring for drift.

5. Trade-offs and wrap-up (5 min)

Name what you would build first (an MVP with a strong eval set), what you'd defer, and the biggest risks.

What separates strong candidates

They start from evaluation, not from picking a model.
They treat the LLM as an unreliable component and design validation, retries and fallbacks around it.
They quantify: tokens per request × requests per day × price per token.
They know when not to use an LLM (a regex or classifier is often better).