When does long context make RAG unnecessary?

30-second answer

For small, stable corpora that fit comfortably in context, stuffing everything in can be simpler. RAG still wins when the corpus is large, changes often, needs per-user permissions, must cite sources, or when cost and latency matter, since long prompts are expensive and slower. Prompt caching narrows the gap for static content.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask about corpus size, update frequency, permissions, citation needs, query volume and latency/cost budget. Framing: "Long context and RAG solve overlapping problems. I'll compare them on cost, quality, freshness and control, then describe when to use each or both."

5–15 min Core explanation

Long context: put the whole document set in the prompt.

Simple: no chunking, indexing or retrieval failures.
The model can reason across the full document (cross-references, global summaries).

RAG: retrieve a few relevant chunks per question.

Scales to corpora far beyond any context window.
Lower cost and latency per query.
Supports freshness, per-user permissions, citations.

15–25 min Deep dive: comparison

Factor	Long context	RAG
Corpus size	Must fit in window	Unlimited
Cost per query	High (all tokens every time), reduced by prompt caching	Low
Latency	Long prefill	Retrieval + short prefill
Global reasoning	Strong	Weak unless designed for it
Pinpoint lookup	Can miss facts in the middle	Strong with good retrieval
Freshness	Rebuild prompt	Update index
Permissions	Must build per-user prompts	Filter at retrieval
Citations	Possible but harder	Natural

Prompt caching changes the economics for stable content: cached prefixes are much cheaper and faster on repeat, making "document in context" viable for a fixed handbook queried many times.

25–32 min Decision guidance

Small (fits comfortably), stable corpus, many queries → long context + prompt caching.
Single document analysis (review this contract, summarize this report) → long context.
Large, changing, multi-tenant or permissioned corpus → RAG.
Mixed needs → RAG to find relevant documents, then load whole documents (not just chunks) into a long context for deep reasoning.

32–37 min Production example

An internal handbook (~60k tokens) assistant moved from RAG to "entire handbook in context with prompt caching". Answers improved for questions spanning several sections, and caching kept cost manageable. Meanwhile the company-wide knowledge assistant (millions of docs, ACLs) stayed on RAG, with a "deep dive" mode that pulls full source documents of the top results into a long context.

37–40 min Likely follow-ups

Will bigger windows make RAG obsolete? Unlikely for large, private, permissioned data; the retrieval problem shifts toward choosing what to load.
How do you evaluate the choice? Same golden set, compare accuracy, latency and cost per query.

What interviewers listen for

Considering cost, permissions and freshness, not just accuracy.
Knowing prompt caching shifts the trade-off.
Hybrid patterns.

Red flags

A blanket answer either way.