Home / RAG & Vector Search

Evaluating RAG systems

Separate retrieval metrics from generation metrics, build a golden set, and use LLM-as-judge carefully.

7 min read

"How do you know your RAG system is good?" is where many candidates stumble. The core idea: evaluate retrieval and generation separately, because they fail differently.

Retrieval metrics

You need a labeled set of queries with their relevant chunks or documents.

  • Recall@k: fraction of relevant items that appear in the top k. The most important metric when a reranker or LLM comes after.
  • Precision@k: fraction of the top k that are relevant.
  • MRR (Mean Reciprocal Rank): average of 1 / rank of the first relevant item.
  • nDCG: rewards relevant items ranked higher, supports graded relevance.

Generation metrics

  • Faithfulness / groundedness: is every claim supported by the retrieved context?
  • Answer relevance: does it actually address the question?
  • Correctness: does it match a reference answer?
  • Citation accuracy: do cited sources really support the sentence that cites them?

Building a golden dataset

  1. Collect real user queries (logs, support tickets) rather than only invented ones.
  2. Add hard cases: multi-hop questions, unanswerable questions, ambiguous ones, exact-ID lookups.
  3. Label relevant sources and reference answers. Synthetic generation by an LLM can bootstrap this, but review a sample by hand.
  4. Keep it versioned and run it in CI on every change to chunking, embeddings, prompts or models.

LLM-as-judge

Using a strong model to grade outputs scales well but has known biases: position bias, preference for longer answers, and self-preference. Mitigate with clear rubrics, pairwise comparisons with randomized order, calibration against a human-labeled sample, and reporting agreement rates.

Online signals

Thumbs up/down, copy events, follow-up rephrasing, escalation to a human, and citation click-through. Offline evals catch regressions; online signals tell you whether users are actually helped.