Home / Question bank / AI System Design

AI System DesignJuniorQuestion 4 of 12

How do you get reliable JSON out of an LLM?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

0–5 min Clarify and frame

Ask what the JSON feeds (UI, database, another service), how complex the schema is, and whether the provider supports structured outputs. Framing: "Use the strongest constraint the platform offers, validate everything, and design for the occasional failure."

5–15 min Core explanation: layers

1. Native structured output / tool calling with a JSON Schema. Many providers can constrain decoding so output always parses and matches the schema.
2. Clear schema design: simple, flat where possible, enums for categorical fields, descriptions per field.
3. Prompt guidance: describe each field, include an example, and say what to do when information is missing (null, not invented).
4. Validation with a schema library (Pydantic, Zod, JSON Schema validator).
5. Repair loop: on failure, retry with the validation error message included.
6. Fallback: after N failures, return a safe error or route to a human.

15–25 min Deep dive: code

from pydantic import BaseModel, Field, ValidationError
from typing import Literal

class Ticket(BaseModel):
    category: Literal["billing", "bug", "feature", "other"]
    priority: Literal["low", "medium", "high"]
    summary: str = Field(max_length=200)
    order_id: str | None = None

def classify(text: str, max_attempts: int = 3) -> Ticket:
    messages = [{"role": "user", "content": build_prompt(text)}]
    for attempt in range(max_attempts):
        raw = call_llm(messages, response_schema=Ticket.model_json_schema())
        try:
            return Ticket.model_validate_json(raw)
        except ValidationError as e:
            messages += [
                {"role": "assistant", "content": raw},
                {"role": "user", "content": f"Invalid output: {e}. Return corrected JSON only."},
            ]
    raise ExtractionFailed(text)

Constrained decoding (grammar-guided generation) masks invalid tokens at each step so output can't violate the grammar. Self-hosted stacks support this with JSON Schema or regex grammars.

25–32 min Trade-offs and pitfalls

• Valid JSON ≠ correct values: validate semantics (e.g. order ID exists) in code.
• Overly complex schemas (deep nesting, many optional fields) reduce quality; split into steps.
• Forcing JSON on reasoning-heavy tasks can hurt quality; let the model reason first (or use built-in reasoning), then emit structured output.
• Low temperature improves consistency for extraction.
• Streaming JSON: parse incrementally only with a tolerant parser, or wait for completion.

32–37 min Production example

A ticket triage service used "please respond in JSON" prompts and saw ~2% parse failures plus invented order IDs. Switching to schema-constrained output, nullable order_id with "null if not present" instructions, and a database check for IDs brought parse failures to zero and eliminated fabricated IDs reaching downstream systems.

37–40 min Likely follow-ups

• How do you monitor it? Track validation failure rate, retry rate, null-field rates by field.
• What if the provider has no structured mode? Function/tool calling, few-shot examples, and validation with repair.

What interviewers listen for

• Native structured outputs + validation + retry + fallback.
• Separating syntactic from semantic validity.

Red flags

• Regex-scraping JSON from free text as the main strategy.