Home / AI System Design

Guardrails, safety and prompt injection

Input and output moderation, prompt-injection defence, and designing agents with least privilege.

9 min read

Safety questions are now standard in AI system design rounds, especially for anything with tools or user-generated content.

Layers of defence

  1. Input checks: policy classification (harassment, self-harm, violence, etc.), PII detection, jailbreak and injection detection, length limits.
  2. Model-level: a carefully written system prompt, a model with good built-in safety training.
  3. Tool and data permissions: the most important layer for agents.
  4. Output checks: policy classification of the response, PII leakage, groundedness for factual answers, format validation.
  5. Human review for high-risk actions and a sample of traffic.
  6. Monitoring and red-teaming to find new failure modes.

Moderation pipeline design

  • Use fast classifiers inline and reserve heavier LLM-based judges for ambiguous cases.
  • Make policies configurable per tenant or product surface: a children's app and a security research tool need different thresholds.
  • For streaming responses, moderate incrementally (on sentence or chunk boundaries) so harmful content can be stopped mid-stream, while balancing latency and false positives.
  • Track precision and recall per category; over-blocking is a real cost.

Prompt injection

Direct injection: the user tries to override instructions ("ignore previous instructions…"). Indirect injection: malicious instructions hidden in content the model reads, such as a web page, email, PDF or tool result. This is the more dangerous one for agents.

There is no complete fix, so design to limit the blast radius:

  • Treat all retrieved and tool content as untrusted data, clearly delimited from instructions.
  • Least privilege: give the agent only the tools and scopes it needs, scoped to the current user.
  • Human confirmation for consequential actions (sending email, payments, deletes).
  • Avoid the "lethal trifecta": an agent with access to private data, exposure to untrusted content, and a way to send data out. Remove at least one.
  • Output channels: block or sanitize links and images in rendered output that could exfiltrate data through URLs.
  • Detection classifiers help but should not be the only control.

Privacy

  • Redact PII before logging; set retention limits.
  • Understand provider data-use and retention terms; use zero-retention options when required.
  • Enforce data residency where regulated.

Interview angle

If asked to design an AI agent that reads email and can send replies, lead with permissions, confirmation steps and injection risk before discussing the model.