What is a reranker and why is it more accurate than the first-stage retriever?

30-second answer

A cross-encoder reranker processes the query and a candidate together, so attention can compare them token by token. A bi-encoder embeds them separately and compares one vector to another, which compresses information. Cross-encoders are too slow to run over the whole corpus, so you retrieve broadly with a bi-encoder or BM25 and rerank a small candidate set.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "Retrieval is two-stage: a fast, broad first stage and an accurate, narrow second stage. I'll compare bi-encoders and cross-encoders architecturally, explain why cross-encoders are more accurate, and cover latency and deployment."

5–15 min Core explanation

Bi-encoder (first stage): encode query and document separately into vectors; relevance = cosine similarity.

Documents are embedded once, offline; queries need one encoding plus an ANN lookup. Scales to billions.
Weakness: all of a document's meaning is compressed into one vector before the query is known.

Cross-encoder (reranker): feed [query ; document] together through a Transformer and output a relevance score.

Every query token can attend to every document token, so it captures exact term matches, negation and specific constraints ("for Python, not Java").
Cost: a full forward pass per (query, document) pair at query time. Impossible over a whole corpus, fine for 20–100 candidates.

15–25 min Deep dive: options and latency

Reranker type	Notes
Cross-encoder models (open or hosted APIs)	Standard choice; tens to low hundreds of ms for 50 candidates on GPU
Late interaction (ColBERT-style)	Token-level vectors with MaxSim; between bi- and cross-encoder in cost and accuracy
LLM-as-reranker	Prompt an LLM to score or sort candidates; flexible but slower and costlier

Latency budgeting: rerank time scales with candidates × length. Truncate long chunks, batch pairs, and pick candidate counts (e.g. 50) by measuring recall vs. latency.

25–32 min Trade-offs

Reranking improves precision at the top, letting you send fewer chunks to the LLM (cheaper, less distraction).
It can't recover documents the first stage missed, so first-stage recall still matters.
Domain shift: off-the-shelf rerankers may underperform on specialised jargon; fine-tune on in-domain pairs if needed.
Score thresholds let you drop low-relevance chunks entirely and trigger "not found" answers.

32–37 min Production example

A support RAG system sent the top 10 vector results to the LLM; answers were often diluted by near-miss articles. Adding a cross-encoder over the top 50 and passing only the top 4 (with a relevance threshold) improved answer accuracy on the eval set, cut prompt tokens by more than half, and added ~80 ms at p50.

37–40 min Likely follow-ups

Why not use the cross-encoder for everything? Cost grows linearly with corpus size per query.
How do you train a reranker? On (query, positive, hard-negative) triples, often mined from logs or first-stage results.
What is a hard negative? A document that looks relevant to the first stage but isn't, the most useful training signal.

What interviewers listen for

Separate vs. joint encoding as the core difference.
Two-stage retrieval economics.
Practical latency and candidate-count tuning.

Red flags

Thinking a reranker improves recall of the overall system without strong first-stage retrieval.