Home / Question bank / AI System Design
What would you log for an LLM request, and what would you avoid logging?
30-second answer
Log model and version, prompt template version, token counts, latency breakdown, retrieved document IDs, tool calls, guardrail decisions, cost, errors and feedback, with a trace ID. Avoid storing raw PII and secrets; redact or tokenize sensitive fields, restrict access, and apply retention limits consistent with privacy commitments.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
Architecture diagram
Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →
Scroll sideways to see the whole diagram.
0–5 min Clarify and frame
Ask about regulatory constraints (GDPR, HIPAA, financial rules), retention requirements, data residency, and who needs access (engineers, support, auditors). Framing: "Log enough metadata to debug, evaluate, attribute cost and audit decisions, while minimizing and protecting sensitive content."
5–15 min Core explanation: what to log
Every request gets a trace ID that links all spans (retrieval, LLM calls, tools, guardrails).
| Category | Fields |
|---|---|
| Context | Feature, tenant, pseudonymous user ID, session/conversation ID, client version |
| Model | Provider, model name and pinned version, decoding parameters |
| Prompt | Template ID and version, prompt hash; full text only in restricted storage |
| Usage | Input tokens, cached input tokens, output tokens, computed cost |
| Latency | Queue time, retrieval, rerank, TTFT, total, per-tool durations |
| Retrieval | Query (rewritten), retrieved document IDs and scores |
| Tools | Tool name, sanitized arguments, status, result size |
| Safety | Guardrail decisions, category scores, policy version |
| Outcome | Errors, retries, fallbacks, stop reason, user feedback, escalation |
15–25 min Deep dive: privacy and storage design
What not to log in plain form: passwords, API keys, payment data, government IDs, health details, and in many contexts raw user messages.
Controls:
- Redaction at ingestion with PII/secret detectors before anything is written.
- Tiered storage:
- metrics and metadata: long retention, broad access;
- traces with sanitized fields: medium retention, engineering access;
- full prompts/responses: short retention, encrypted, access-controlled and audited, possibly sampled.
- Deletion: propagate user deletion requests to logs, caches and eval datasets derived from logs.
- Residency: keep logs in the required region.
- Vendor settings: understand your LLM provider's retention and use zero-retention options where needed.
Standards: emerging OpenTelemetry semantic conventions for generative AI spans let you use vendor-neutral tracing backends.
25–32 min Using the data
- Dashboards: latency percentiles, error rates, cost per feature/tenant, guardrail trigger rates, feedback rates.
- Alerts: cost anomalies, error spikes, latency regressions after deploys.
- Debugging: follow a bad answer's trace from retrieval to output.
- Evaluation: turn redacted failures into new eval cases.
- Audit: show which policy version and model produced a decision.
32–37 min Production example
After a release, thumbs-down rate rose for one feature. Because traces recorded prompt version and retrieved document IDs, the team saw the new prompt produced shorter rewritten queries that retrieved worse documents. They rolled back in minutes and added the failing cases to the eval set. Full transcripts weren't needed, since metadata was enough.
37–40 min Likely follow-ups
- Should you sample? Keep 100% of metadata; sample full content.
- How do you let support see conversations safely? Role-based access, redaction, audit logs, time-limited access.
What interviewers listen for
- Comprehensive, structured metadata with trace IDs.
- Concrete privacy controls and retention tiers.
- How the logs feed debugging and evals.
Red flags
- Logging raw prompts with secrets indefinitely, or logging too little to debug.