Home / RAG & Vector Search

The RAG pipeline end to end

Ingestion, chunking, embedding, retrieval, reranking and generation, plus where each step goes wrong.

11 min read

Retrieval-Augmented Generation answers a question by first finding relevant text, then giving that text to the model as context. It is the default way to make an LLM answer from private or fresh data, and it shows up in nearly every applied-AI interview.

Two pipelines

Offline (indexing):

  1. Load documents (PDFs, HTML, tickets, code, wikis).
  2. Clean and parse: strip boilerplate, keep structure such as headings and tables.
  3. Chunk into retrievable units.
  4. Embed each chunk with an embedding model.
  5. Store vectors plus metadata (source, section, date, permissions) in an index.

Online (query time):

  1. Optionally rewrite the query (expand acronyms, split multi-part questions, add conversation context).
  2. Retrieve top-k candidates (vector, keyword, or both).
  3. Rerank candidates with a more accurate model.
  4. Assemble a prompt with the best chunks and instructions to cite and to admit missing information.
  5. Generate, then optionally verify citations.

Chunking

Chunking is the most underrated lever in RAG quality.

Strategy When it works
Fixed-size with overlap (e.g. 300–800 tokens, 10–20% overlap) Simple baseline for uniform prose
Structure-aware (by heading, section, function) Docs, manuals, code. Usually better
Semantic (split where embedding similarity drops) Long unstructured text
Parent-child / small-to-big Retrieve small precise chunks, pass the larger parent section to the model

Chunks that are too small lose context ("it" with no referent). Chunks that are too large dilute the embedding and waste context window. Contextual chunk headers (prepending the document title and section path to each chunk before embedding) are a cheap, effective improvement.

Retrieval: dense, sparse, hybrid

  • Dense retrieval embeds the query and finds nearest chunk vectors. Good at paraphrase and meaning.
  • Sparse retrieval (BM25) scores keyword overlap. Good at exact terms: error codes, product names, IDs.
  • Hybrid runs both and merges results, commonly with Reciprocal Rank Fusion: score(d) = Σ 1 / (k + rank_i(d)) with k ≈ 60. Hybrid is a strong default.

Reranking

Bi-encoder embeddings are fast because query and document are encoded separately, but they are less precise. A cross-encoder reranker reads the query and each candidate together and scores relevance more accurately. Typical pattern: retrieve 50–100 cheaply, rerank to the top 5–10.

Generation

  • Put instructions first and retrieved context in clearly delimited blocks with source IDs.
  • Ask for citations by source ID and tell the model to say when the context does not contain the answer.
  • Be aware of lost in the middle: models attend best to the start and end of long contexts, so ordering matters.

Common failure modes

Symptom Likely cause Fix
Right doc exists, not retrieved Bad chunking, vocabulary mismatch Hybrid search, query rewriting, better chunks
Retrieved but answer still wrong Too many distractors, poor ordering Reranking, fewer but better chunks
Confident answer with no support Model ignores context Stronger grounding instructions, citation checks
Stale or forbidden content No metadata filtering Filter by date, tenant and ACL at query time

Advanced patterns

  • Multi-query / HyDE: generate several query variants or a hypothetical answer and retrieve with those.
  • Agentic RAG: let the model decide when and what to search, iterating until it has enough.
  • GraphRAG: build an entity graph to answer questions that span many documents.
  • Long context vs. RAG: larger context windows reduce the need for aggressive retrieval, but RAG still wins on cost, latency, freshness, permissions and citations.