How do you handle a conversation that exceeds the context window?

30-second answer

Count tokens before sending. Options: drop the oldest turns while keeping the system prompt and recent turns; summarize older history into a running summary; move long-term facts into retrieval or a memory store; truncate large tool outputs. Choose based on what information the task actually needs.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask what's filling the context: long chat history, big documents, or large tool outputs in an agent? What information must persist across the conversation (user preferences, decisions, IDs)? Framing: "Count tokens, decide what matters, and compress or move the rest out of the window before you hit the limit."

5–15 min Core explanation: strategies

Count tokens with the model's tokenizer or the provider's token-counting API before every call. Reserve room for the response (max_tokens) plus a safety margin.
Sliding window: keep the system prompt and the last N turns. Simple and predictable, but early facts vanish.
Running summary: when history passes a threshold (e.g. 70% of budget), summarize older turns into a compact message.
Long-term memory store: extract durable facts into a store and retrieve relevant ones per turn.
Externalize large artifacts: keep big documents and tool outputs outside the context, referenced by ID, with a tool to re-read them when needed.
Prompt caching: reduces cost and latency of a long stable prefix but doesn't raise the limit.

15–25 min Deep dive: a context layout

[system prompt + tool definitions]        stable → cacheable
[retrieved long-term memory facts]        small, relevant only
[summary of turns 1..k]                   compressed history
[recent turns k+1..n verbatim]            full detail where it matters most
[current user message]

def build_messages(conv, budget):
    msgs = conv.recent_turns()
    while count_tokens(SYSTEM, conv.summary, msgs) > budget * 0.7 and len(msgs) > 4:
        oldest = msgs[:2]
        conv.summary = summarize(conv.summary, oldest)   # keep decisions, IDs, constraints
        msgs = msgs[2:]
    return [summary_message(conv.summary)] + msgs

What a good summary keeps: decisions made, open questions, user constraints and preferences, identifiers (order numbers, file paths), and anything the user explicitly asked to remember. Keep critical facts in a structured list rather than prose so they survive repeated summarization.

25–32 min Trade-offs

Strategy	Pros	Cons
Sliding window	Simple, cheap	Forgets early facts
Summarization	Retains gist	Lossy; summary errors compound; extra LLM call
Memory retrieval	Scales indefinitely	Retrieval misses
Externalized artifacts	Keeps context lean	Model must decide to re-read; more tool calls
Bigger context model	Fewer compressions	Higher cost per call; attention dilution

Tool outputs are a frequent culprit in agents: return filtered, structured results instead of raw dumps.

32–37 min Production example

A coding assistant's long sessions failed with context-limit errors because whole files were pasted into history repeatedly. Changes: file contents kept out of history with path references and a read_file tool; old turns summarized into a list of decisions and modified files; token counting before each call with summarization at 70%. Sessions became effectively unbounded and answer quality on long-session evals stayed stable.

37–40 min Likely follow-ups

How do you test this? Long synthetic conversations that introduce facts early and ask about them later; measure recall of those facts.
What about multi-user shared context? Scope memory per user and per permission set.

What interviewers listen for

Token counting with reserved output budget.
Multiple strategies with their trade-offs and what must persist.

Red flags

Truncating by characters, or letting requests fail at the limit.