How do you evaluate a RAG system?

30-second answer

Separate retrieval from generation. Retrieval: recall@k, MRR, nDCG on a labeled query set. Generation: faithfulness to context, answer relevance, correctness against references, citation accuracy, often with a calibrated LLM-as-judge. Include unanswerable and multi-hop cases, version the set, run it in CI, and complement with online feedback signals.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "RAG has two failure points, retrieval and generation, so I'll evaluate them separately, then end to end, offline and online."

5–15 min Core explanation: what to measure

Retrieval (needs labelled relevant passages per query):

Recall@k: did relevant passages make the top k? Most important when a reranker/LLM follows.
Precision@k: how much of the top k is relevant.
MRR: how high the first relevant result ranks.
nDCG: graded relevance with position discount.

Generation:

Faithfulness/groundedness: every claim supported by the provided context.
Answer relevance: addresses the actual question.
Correctness: matches a reference answer.
Citation precision/recall: cited sources really support claims; important claims are cited.
Abstention quality: says "not found" for unanswerable questions, but not for answerable ones.

15–25 min Deep dive: building the eval set

Source questions from real logs, support tickets, or subject-matter experts. Synthetic questions generated from chunks can bootstrap, but they're biased toward easy lexical matches.
Cover hard cases: multi-hop, comparisons, exact IDs, ambiguous questions, unanswerable questions, stale-vs-new conflicts.
Label relevant passages and reference answers; record difficulty and category.
Version the set; keep a held-out portion to avoid overfitting prompts to it.

Size: 100–300 well-chosen examples often beats thousands of synthetic ones.

LLM-as-judge for faithfulness and correctness:

Use explicit rubrics and require the judge to quote supporting text.
Calibrate against human labels on a sample; report agreement.
Mitigate position and length bias (randomize order for pairwise comparisons).

25–32 min Process

Run evals in CI on any change to chunking, embeddings, retriever, reranker, prompt or model.
Report per-category scores; block on regressions.
Use failure analysis: for each wrong answer, label whether retrieval or generation failed. This tells you where to invest.
Track latency and cost alongside quality.

32–37 min Production example

A team's answer accuracy plateaued. Failure analysis on 150 errors showed 60% were retrieval misses (mostly exact product codes), 25% were generation ignoring context, 15% were bad labels. Adding hybrid search fixed most retrieval misses; tightening the prompt and adding a citation verifier addressed generation. Without separating the two, they'd have kept swapping LLMs.

37–40 min Likely follow-ups

What online signals do you use? Thumbs, rephrased follow-ups, escalation to humans, citation clicks, session abandonment.
How do you A/B test RAG changes? Randomize users, compare task success and feedback, guardrail on latency and cost.
Can you evaluate without labels? Partially: faithfulness checks need only context and answer; recall needs labels.

What interviewers listen for

Separating retrieval from generation metrics.
Realistic eval set construction including unanswerable questions.
Judge calibration and failure analysis.

Red flags

"We'd check a few answers manually."
Only measuring end-to-end accuracy.