Home / RAG & Vector Search

Embeddings and vector indexes

Similarity metrics, exact vs. approximate search, HNSW and IVF, quantization, and filtering.

9 min read

Similarity metrics

  • Cosine similarity: angle between vectors, ignores length. The usual choice for text embeddings.
  • Dot product: equals cosine when vectors are normalized to unit length, and is faster.
  • Euclidean (L2): straight-line distance. Ranking matches cosine on normalized vectors.

Always use the metric the embedding model was trained with.

Exact (brute-force) k-NN compares the query with every vector: O(N·d). That is fine for tens or hundreds of thousands of vectors and on GPUs, but too slow at hundreds of millions. Approximate Nearest Neighbour (ANN) indexes trade a little recall for large speedups.

HNSW (Hierarchical Navigable Small World)

A multi-layer graph. Upper layers are sparse "highways"; the bottom layer contains every vector. Search starts at the top and greedily moves toward the query, descending layers.

  • Key parameters: M (links per node), efConstruction (build quality), efSearch (query-time breadth).
  • Excellent recall and latency. Memory-heavy, since it stores vectors plus graph edges. Supports incremental inserts.

IVF (Inverted File)

Cluster vectors with k-means into nlist centroids. At query time, search only the nprobe nearest clusters.

  • Lower memory than HNSW and easy to combine with compression.
  • Needs training on representative data. Recall depends on nprobe.

Compression

  • Product Quantization (PQ): split each vector into sub-vectors and replace each with a codebook ID. Large memory savings, some accuracy loss. IVF-PQ is a classic billion-scale combination.
  • Scalar / binary quantization: store float32 as int8 or single bits, then re-score top candidates with full-precision vectors.

Filtering

Real queries need constraints: tenant, permissions, date range, language.

  • Post-filtering: retrieve top-k, then filter. Can return too few results.
  • Pre-filtering: filter first, then search. Can be slow or break graph connectivity in HNSW.
  • Filtered/hybrid index support in the database is the practical answer. Ask how your database handles it.

Choosing a store

Option Fits when
pgvector in Postgres You already run Postgres and have up to a few million vectors; you want joins and transactions
Dedicated vector DB (managed or self-hosted) Large scale, heavy filtering, high QPS, hybrid search built in
Search engine with vector support You already need strong keyword search and facets
In-process library (e.g. FAISS) Offline batch jobs, research, single-node services

Embedding model choice

  • Evaluate on your data with a retrieval metric, not just public leaderboards.
  • Consider dimension (storage and speed), max input length, multilingual needs, and cost.
  • Changing embedding models means re-embedding everything. Version your index and plan for backfills.