Home / Question bank
Question bank
48 AI interview questions. Each has a 30-second answer and a full walkthrough paced for a 40-minute interview round. Answer out loud first, then compare.
What is a token, and why do LLMs use subword tokenization instead of words or characters?
30-second answer
A token is an integer ID from the model's fixed vocabulary. Word-level vocabularies are huge and can't handle unseen words; character-level sequences are very long, which is expensive for attention. Subword schemes like BPE balance the two: common words are single tokens, rare words split into reusable pieces, and byte fallback means any input can be encoded.
Full 40-minute answer →Explain scaled dot-product attention. Why divide by √d_k?
30-second answer
Each token is projected to query, key and value vectors. Scores are QKᵀ, normalized row-wise with softmax, then used to take a weighted sum of V. Dot products grow in magnitude with dimension; without scaling, softmax saturates into near one-hot outputs with tiny gradients. Dividing by √d_k keeps the variance of the scores roughly constant.
Full 40-minute answer →What is the KV cache and what problem does it solve?
30-second answer
In autoregressive decoding, keys and values for earlier tokens don't change. Caching them means each new token computes only its own projections and attends to the cache, turning per-step cost from re-processing the whole sequence into a single-token step. The trade-off is memory: the cache grows with layers, KV heads, head dimension and sequence length, and often limits batch size in serving.
Full 40-minute answer →Why is attention O(n²), and what are common ways to handle long context?
30-second answer
Every token attends to every other token, so the score matrix is n×n. Mitigations: FlashAttention (exact, IO-aware, avoids materializing the matrix), sliding-window or local attention, sparse patterns, linear-attention or state-space alternatives, and at the system level, retrieval so you don't need the whole corpus in context. Positional schemes like RoPE with scaling help models extrapolate to longer inputs.
Full 40-minute answer →What's the difference between MHA, MQA and GQA?
30-second answer
Multi-head attention gives each head its own K and V. Multi-query attention shares a single K/V across all heads, shrinking the KV cache dramatically at some quality cost. Grouped-query attention shares K/V within groups of heads, a middle ground used by many current models to cut memory and boost decode throughput.
Full 40-minute answer →Describe prefill vs. decode. Which one dominates latency for long answers?
30-second answer
Prefill processes the whole prompt in parallel and is compute-bound; it determines time-to-first-token. Decode generates one token per forward pass and is memory-bandwidth-bound; it determines tokens/second. For long outputs, decode dominates total latency, which is why output length is the first lever for speed.
Full 40-minute answer →When would you choose fine-tuning over RAG, and vice versa?
30-second answer
Use RAG when the model needs fresh, private, or citable knowledge, or when data changes often. Use fine-tuning to change behaviour: consistent format or tone, domain-specific style, a narrow skill, or to shorten long prompts at high volume. Fine-tuning is unreliable for injecting facts. Often the best answer is both: fine-tune for behaviour, retrieve for knowledge, and start with prompting because it's cheapest to iterate.
Full 40-minute answer →How does LoRA work and why is it popular?
30-second answer
LoRA freezes the pretrained weight W and learns a low-rank update ΔW = BA with small rank r. Only A and B are trained, cutting trainable parameters and optimizer memory by orders of magnitude. Adapters are small, can be swapped per task or tenant, and can be merged into W so inference has no extra latency. QLoRA adds a 4-bit quantized base to fit larger models on a single GPU.
Full 40-minute answer →Compare RLHF and DPO.
30-second answer
RLHF trains a reward model on human preference comparisons, then optimizes the policy against it with RL (commonly PPO) plus a KL penalty to the reference model. DPO derives a closed-form objective that optimizes the policy directly on preference pairs, with no separate reward model or RL loop. DPO is simpler and more stable; RLHF can be more flexible, for example when rewards come from sources other than pairwise preferences.
Full 40-minute answer →What do temperature and top-p do?
30-second answer
Temperature divides the logits before softmax: below 1 sharpens the distribution (more deterministic), above 1 flattens it (more diverse). Top-p samples only from the smallest set of tokens whose cumulative probability exceeds p, cutting off the unreliable tail. For extraction or classification, use low temperature; for brainstorming, higher.
Full 40-minute answer →Why do LLMs hallucinate, and how do you reduce it in a product?
30-second answer
Pre-training optimizes for plausible continuations, not truth, so the model will fill gaps fluently. In products: ground answers with retrieval, require citations and verify them, allow and reward 'I don't know', use lower temperature for factual tasks, constrain output formats, and measure faithfulness with evals rather than spot checks.
Full 40-minute answer →What is a Mixture-of-Experts model?
30-second answer
MoE replaces the dense feed-forward layer with many expert FFNs and a learned router that sends each token to a small number of them (e.g. top-2). Total parameters grow a lot while per-token compute stays modest. Challenges include load balancing across experts (auxiliary losses), higher memory needs since all experts must be resident, and more complex distributed serving.
Full 40-minute answer →What is speculative decoding?
30-second answer
A small, fast draft model proposes several tokens; the large model verifies them in a single forward pass and accepts the longest prefix consistent with its own distribution, correcting the first rejected token. Because verification is parallel, you get several tokens per large-model pass. With the standard acceptance rule the output distribution is unchanged, so it's a pure latency win when the draft agrees often.
Full 40-minute answer →Walk me through how you'd build a RAG system over a company's internal docs.
30-second answer
Offline: load and parse docs keeping structure, chunk by section with overlap, prepend title/section headers, embed, store with metadata (source, date, ACL). Online: rewrite the query with conversation context, run hybrid retrieval (BM25 + vectors) with permission filters, rerank top ~50 to ~5, build a prompt with delimited, ID-tagged chunks and instructions to cite and to say when the answer isn't present, then generate. Wrap it with an eval set measuring recall@k and faithfulness, plus logging and feedback.
Full 40-minute answer →How do you choose chunk size?
30-second answer
Empirically, against a retrieval eval. Start with structure-aware chunks of a few hundred tokens with modest overlap. Too small loses context; too large dilutes the embedding and wastes prompt space. Consider parent-child retrieval: match on small chunks, pass the larger parent section to the model. Measure recall@k and answer quality across a few settings rather than guessing.
Full 40-minute answer →Why use hybrid search instead of only vector search?
30-second answer
Dense embeddings capture meaning and paraphrase but are weak on exact tokens such as error codes, SKUs, names and rare jargon. BM25 excels at exact matches. Combining them (often with Reciprocal Rank Fusion) improves recall across both query types at little cost.
Full 40-minute answer →What is a reranker and why is it more accurate than the first-stage retriever?
30-second answer
A cross-encoder reranker processes the query and a candidate together, so attention can compare them token by token. A bi-encoder embeds them separately and compares one vector to another, which compresses information. Cross-encoders are too slow to run over the whole corpus, so you retrieve broadly with a bi-encoder or BM25 and rerank a small candidate set.
Full 40-minute answer →Explain HNSW and its main parameters.
30-second answer
HNSW builds a layered proximity graph: sparse upper layers for long jumps, a dense bottom layer with every vector. Search greedily walks from an entry point toward the query, descending layers. M controls edges per node (recall and memory), efConstruction controls build quality, efSearch controls query-time candidate list size (recall vs. latency). It gives high recall at low latency but uses significant memory.
Full 40-minute answer →HNSW vs. IVF-PQ: when would you choose each?
30-second answer
HNSW for high recall and low latency when the vectors fit in memory and you need incremental inserts. IVF-PQ when scale or memory cost dominates (hundreds of millions to billions of vectors): IVF limits the search to a few clusters and PQ compresses vectors heavily, at some accuracy cost, often with re-ranking on full vectors.
Full 40-minute answer →How do you enforce document permissions in RAG?
30-second answer
Store ACL metadata (tenant, groups, document IDs) with each chunk and filter at retrieval time using the caller's identity, never relying on the LLM to withhold information. Use a vector store that supports efficient filtered search, keep ACLs in sync with the source system, and test for leakage in evals. Per-tenant indexes are an option for strong isolation.
Full 40-minute answer →How do you evaluate a RAG system?
30-second answer
Separate retrieval from generation. Retrieval: recall@k, MRR, nDCG on a labeled query set. Generation: faithfulness to context, answer relevance, correctness against references, citation accuracy, often with a calibrated LLM-as-judge. Include unanswerable and multi-hop cases, version the set, run it in CI, and complement with online feedback signals.
Full 40-minute answer →What is 'lost in the middle' and how do you mitigate it?
30-second answer
Models tend to use information at the beginning and end of a long context better than information in the middle. Mitigate by sending fewer, better-reranked chunks, placing the most relevant ones at the edges, and keeping prompts focused rather than stuffing the context window.
Full 40-minute answer →Your embedding provider releases a better model. How do you migrate?
30-second answer
Embeddings from different models aren't comparable, so you must re-embed the corpus. Build a new versioned index in parallel, backfill in batches, evaluate it against the old one on the golden set, dual-write new documents to both during transition, then shift traffic gradually and keep the old index for rollback.
Full 40-minute answer →When does long context make RAG unnecessary?
30-second answer
For small, stable corpora that fit comfortably in context, stuffing everything in can be simpler. RAG still wins when the corpus is large, changes often, needs per-user permissions, must cite sources, or when cost and latency matter, since long prompts are expensive and slower. Prompt caching narrows the gap for static content.
Full 40-minute answer →What is HyDE?
30-second answer
Hypothetical Document Embeddings: have the LLM write a plausible answer to the query, embed that, and search with it. A hypothetical answer often lies closer to real answer passages in embedding space than a short question does. It costs an extra LLM call and can drift if the hypothetical answer is wrong.
Full 40-minute answer →Design a customer-support chatbot that answers from a company knowledge base.
30-second answer
Clarify scale, latency, languages and escalation requirements. Metrics: resolution/deflection rate, groundedness, CSAT, p95 latency, cost per conversation. Architecture: ingestion pipeline (parse, chunk by section, embed, index with metadata), orchestrator with input guardrails, hybrid retrieval + rerank, an LLM prompted to answer only from cited context and hand off when unsure, output checks, streaming. Deep dives: evals (golden set with unanswerable cases), human handoff, freshness of docs, caching common questions, and monitoring with feedback loops.
Full 40-minute answer →How would you reduce the latency of an LLM feature that currently takes 8 seconds?
30-second answer
First measure the breakdown: retrieval, queueing, time-to-first-token, decode. Then: stream output; cut output length; use a smaller or faster model for some or all steps; parallelize retrieval and tool calls; trim the prompt and use prompt caching for static prefixes; cache frequent responses. If self-hosting, use continuous batching, quantization and speculative decoding.
Full 40-minute answer →Your LLM bill doubled last month. How do you investigate and reduce it?
30-second answer
Break cost down by feature, model, tenant and input vs. output tokens using request logs. Look for prompt bloat (too many retrieved chunks, long histories), runaway agent loops, retries, or traffic growth. Reduce with routing to cheaper models, prompt caching, shorter outputs, conversation summarization, response caching, batch APIs for offline work, and budgets/limits per user or tenant.
Full 40-minute answer →How do you get reliable JSON out of an LLM?
30-second answer
Use the provider's structured output or tool-calling mode with a JSON schema where available. Give a clear schema and examples. Validate every response with a schema validator; on failure, retry with the validation error included. Keep schemas simple, and design downstream code to handle a final failure gracefully.
Full 40-minute answer →What is prompt injection and how do you defend against it?
30-second answer
It's when input causes the model to follow attacker instructions instead of the developer's. Indirect injection arrives through content the model reads (web pages, emails, documents). Defences limit impact rather than eliminate it: treat external content as data, least-privilege tools scoped to the user, human confirmation for consequential actions, block data exfiltration channels like auto-loaded URLs, injection classifiers as one layer, and red-team testing.
Full 40-minute answer →Design a content moderation service that inspects LLM outputs in real time.
30-second answer
Requirements: categories and per-tenant policies, latency budget, streaming support. Architecture: a policy service holding configurable thresholds, a fast tier of lightweight classifiers inline, a slower LLM-judge tier for borderline cases, and streaming inspection on chunk/sentence boundaries with the ability to halt and replace the response. Log decisions with reasons for audit, measure precision/recall per category on labeled sets, support appeals and human review queues, and fail safely (decide fail-open vs. fail-closed per surface).
Full 40-minute answer →How would you evaluate a new model before switching production traffic to it?
30-second answer
Run it on the offline golden set covering quality, safety and format adherence, and compare against the current model with the same prompts (adjusting prompts if needed). Check latency and cost. Then shadow traffic or a small canary with online metrics and guardrail rates, followed by a staged rollout with a quick rollback path.
Full 40-minute answer →When should you NOT use an LLM?
30-second answer
When a deterministic rule, regex, lookup or small classifier solves the problem reliably; when latency or cost budgets are tiny and volume huge; when outputs must be exactly reproducible or provably correct; or when you can't tolerate any unverified errors and can't add verification. LLMs are best where inputs are messy and some fuzziness is acceptable or checkable.
Full 40-minute answer →Explain semantic caching and its risks.
30-second answer
Store past query embeddings with their answers; for a new query, return a cached answer if a previous query is similar enough. It saves cost and latency for repetitive traffic. Risks: near-identical questions can need different answers ("cancel my order" vs. "don't cancel my order"), personalization and permission differences, and stale answers. Use conservative thresholds, scope caches by user/tenant, add TTLs, and evaluate false-hit rates.
Full 40-minute answer →How would you design a model router?
30-second answer
Classify each request by difficulty or type with a cheap model or heuristics, then send easy ones to a small model and hard ones to a large model. Alternatively, cascade: try the small model and escalate if a verifier or confidence signal is low. Measure the quality/cost frontier on an eval set, include fallbacks for provider outages, and monitor routing distribution for drift.
Full 40-minute answer →What would you log for an LLM request, and what would you avoid logging?
30-second answer
Log model and version, prompt template version, token counts, latency breakdown, retrieved document IDs, tool calls, guardrail decisions, cost, errors and feedback, with a trace ID. Avoid storing raw PII and secrets; redact or tokenize sensitive fields, restrict access, and apply retention limits consistent with privacy commitments.
Full 40-minute answer →What is the 'lethal trifecta' for AI agents?
30-second answer
An agent that combines access to private data, exposure to untrusted content, and a channel to communicate externally. Untrusted content can inject instructions that make the agent read private data and send it out. The mitigation is architectural: remove at least one leg, for example by blocking outbound channels or requiring human approval for them.
Full 40-minute answer →How does function calling / tool use work under the hood?
30-second answer
You send tool definitions (name, description, JSON schema). The model is trained to emit a structured tool call when appropriate. Your application parses it, executes the function, and returns the result as a tool-result message. The model then continues, possibly calling more tools, until it produces a final answer. The model never runs code itself.
Full 40-minute answer →Implement an agent loop. What safeguards do you add?
30-second answer
Loop: call the model with tools and history; if it requests tools, validate arguments, authorize against the user's permissions, execute (in parallel when independent), append results, repeat; stop on a final answer. Safeguards: max iterations and token budget, timeouts, errors returned as tool results, idempotency for side effects, human confirmation for destructive actions, logging/tracing of every step.
Full 40-minute answer →When would you use a workflow (chained prompts) instead of an autonomous agent?
30-second answer
When the steps are known in advance. Workflows are predictable, cheaper, faster and easier to test and debug. Use an agent only when the path genuinely depends on intermediate results and can't be enumerated, and accept the extra cost, latency and risk.
Full 40-minute answer →How do you handle a conversation that exceeds the context window?
30-second answer
Count tokens before sending. Options: drop the oldest turns while keeping the system prompt and recent turns; summarize older history into a running summary; move long-term facts into retrieval or a memory store; truncate large tool outputs. Choose based on what information the task actually needs.
Full 40-minute answer →How do you test code that calls an LLM?
30-second answer
Unit-test the surrounding code with a mocked client (prompt rendering, parsing, retries, tools). Validate output against schemas. Maintain an eval set graded by exact match, programmatic checks or a calibrated LLM judge, run it in CI on prompt/model changes, and look at pass rates over repeated runs. Add online monitoring and user feedback.
Full 40-minute answer →What is few-shot prompting and when does it help?
30-second answer
Including a few input–output examples in the prompt. It helps most with output format, style and tricky edge cases that are hard to describe. Use diverse examples so the model generalizes rather than copying, and be aware that examples cost tokens on every request (prompt caching helps).
Full 40-minute answer →How would you stream an LLM response to a web client?
30-second answer
Call the provider's streaming endpoint (server-sent events) from the backend, never exposing API keys to the browser. Forward text deltas to the client via SSE or WebSockets, flush promptly, handle client disconnects by cancelling the upstream request, buffer tool-call arguments until complete, and record usage and errors from the final events.
Full 40-minute answer →What is MCP?
30-second answer
The Model Context Protocol is an open standard for connecting AI applications to tools and data through a client–server interface. A server exposes tools, resources or prompts; any MCP-capable client can use them. It reduces one-off integrations. Security considerations include trusting servers, scoping credentials, and treating server outputs as untrusted content.
Full 40-minute answer →How do you handle rate limits from an LLM provider at scale?
30-second answer
Client-side: exponential backoff with jitter honoring Retry-After. System-side: a token-aware rate limiter or queue in front of the provider, prioritizing interactive over batch traffic, per-tenant quotas, request batching where supported, prompt caching to reduce token load, and fallbacks to other models or regions. Monitor utilization against provisioned limits and request increases ahead of launches.
Full 40-minute answer →Your prompt change improved one test case but you're unsure it's better overall. What do you do?
30-second answer
Run the full eval set, ideally multiple times to account for non-determinism, and compare per-category scores against the baseline. Check for regressions in safety and format adherence. If the offline result is positive, roll out behind a flag or A/B test and watch online metrics before making it the default.
Full 40-minute answer →How should an application store and use LLM API keys?
30-second answer
Keep keys on the server in a secrets manager or environment configuration, never in client code or source control. Use separate keys per environment and service, rotate them, restrict scopes and set spending limits where possible, and proxy client requests through your backend with your own auth and rate limits.
Full 40-minute answer →No questions match those filters.