How do you choose chunk size?

30-second answer

Empirically, against a retrieval eval. Start with structure-aware chunks of a few hundred tokens with modest overlap. Too small loses context; too large dilutes the embedding and wastes prompt space. Consider parent-child retrieval: match on small chunks, pass the larger parent section to the model. Measure recall@k and answer quality across a few settings rather than guessing.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask what the documents look like (manuals with headings? chat logs? code? legal contracts?) and what questions users ask (pinpoint facts vs. summaries). Framing: "Chunk size trades precision against context. I'll explain the forces, the strategies, and how I'd choose empirically."

5–15 min Core explanation

A chunk is the unit you embed and retrieve. Its size matters because:

Too small (e.g. single sentences): embeddings are precise but chunks lose context ("it", "the above"). The model may receive fragments that don't answer the question.
Too large (e.g. whole pages): one embedding must represent many topics, so similarity is diluted and recall drops. Each chunk consumes more prompt tokens, so you fit fewer distinct sources.
Overlap (10–20%) avoids cutting an answer across a boundary, at the cost of index size and duplicate results.

Typical starting points: 300–800 tokens for prose, function or class level for code, one Q&A pair for FAQs.

15–25 min Deep dive: strategies

Strategy	How	Best for
Fixed size + overlap	Split every N tokens	Uniform prose; baseline
Recursive / structural	Split on headings, then paragraphs, then sentences	Docs, wikis, manuals
Semantic	Split where embedding similarity between sentences drops	Long unstructured text
Document-specific	By function (code), clause (contracts), message thread (chat)	Domain formats
Parent–child (small-to-big)	Retrieve small chunks; return their parent section	Precision and context together
Sentence window	Retrieve a sentence; expand to neighbouring sentences	Pinpoint facts

Enhancements that often beat tuning size:

Contextual headers: prepend document title and section path before embedding.
Contextual summaries: have an LLM write a one-line description of how the chunk fits its document, prepended at index time.
Keep tables intact with header rows; don't split code mid-function.

25–32 min Choosing empirically

Build a labelled query set (50–200 questions with the relevant passages marked).
Try 3–4 configurations (e.g. 256/512/1024 tokens, structural vs. fixed, with/without headers).
Measure recall@k for retrieval and end-to-end answer quality.
Also measure cost: index size, tokens per prompt.
Choose the simplest configuration near the top; re-check when the corpus changes.

Different document types can use different chunkers in the same index; store the chunker version in metadata.

32–37 min Production example

An engineering runbook assistant used 1,000-token fixed chunks and often returned the right page but not the right step. Switching to heading-based chunks of ~400 tokens with breadcrumbs (Service X > Incident response > Restart procedure) and parent-section expansion raised recall@5 on their eval set and made answers more specific, while prompts got shorter.

37–40 min Likely follow-ups

Does the embedding model's max length matter? Yes; text beyond it is truncated silently. Chunk below it.
What about late chunking? Embed the whole document with a long-context embedding model, then pool token embeddings per chunk so each chunk carries document context.
How often should you re-chunk? When eval scores drop, content formats change, or you change embedding models.

What interviewers listen for

The precision vs. context trade-off explained concretely.
Structure-aware and parent–child strategies.
An empirical, eval-driven method rather than a magic number.

Red flags

A single number with no justification.
Ignoring document structure and tables.