Home / AI Architecture & System Design / Agents & LLM Applications

Agents & LLM ApplicationsMidQuestion 2 of 9

Implement an agent loop. What safeguards do you add?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Functional requirements

"Implement an agent loop" is easy to write in ten lines and hard to run in production. Agree on what the agent does and what can go wrong.

You What does the agent do, and what tools does it have?

Interviewer It's an operations assistant for on-call engineers. Tools: search logs, query metrics, read runbooks, restart a service, and open an incident ticket.

You Which of those have side effects?

Interviewer Restarting a service and opening a ticket.

You How long do tasks run?

Interviewer Usually a few minutes. "Why is checkout latency up?" might take 10–20 tool calls.

You Who's allowed to restart services?

Interviewer Only on-call engineers, and only services their team owns.

5–10 min Non-functional requirements and estimates

Back-of-envelope estimates:

• Context growth: each step adds the model's message plus tool results. If a log search returns 3,000 tokens and a task takes 15 steps, history grows to ~45,000+ tokens by the end. Re-sent every step, that's roughly 15 × 25,000 ≈ 375,000 input tokens per task on average. This is why tool output must be filtered and history compacted.
• With compaction (tool results trimmed to ~500 tokens, older steps summarized), average context stays ~8,000 tokens: ~120,000 input tokens per task, roughly a 3× saving.
• Volume: 2,000 tasks/day across on-call teams; ~30,000 model calls/day. Small enough that correctness and safety dominate over throughput.

10–18 min High-level design

You The loop runs in an orchestrator that owns state, budgets and the trace. All tool calls go through one executor that validates, authorizes, asks for confirmation when needed, and returns structured results or structured errors.

The loop:

def run_task(user, task, budget):
    state = load_or_create_state(task.id)          # resumable
    while True:
        if budget.exceeded(state):
            return finish(state, reason="budget", partial=summarize_findings(state))
        resp = llm.create(system=SYSTEM, tools=tool_specs_for(user),
                          messages=compact(state.messages), max_tokens=2048)
        state.record(resp)                          # tokens, cost, latency
        if resp.stop_reason == "end_turn":
            return finish(state, answer=resp.text)
        calls = [b for b in resp.content if b.type == "tool_use"]
        results = run_parallel(calls, lambda c: execute(user, c, state))
        state.messages += [assistant(resp.content), tool_results(results)]
        stream_status(user, calls, results)
        detect_loops(state)                         # same call repeated → nudge or stop
        save_state(state)

The executor:

def execute(user, call, state):
    spec = TOOLS[call.name]
    try:
        args = spec.schema.validate(call.input)               # types, ranges, enums
        authorize(user, call.name, args)                      # e.g. service owned by user's team
        if spec.side_effect:
            if not confirm(user, describe(call.name, args)):  # shows exact action
                return error(call.id, "User declined this action.")
            args["idempotency_key"] = f"{state.task_id}:{call.id}"
        out = run_with_timeout(spec.fn, args, spec.timeout_s)
        return ok(call.id, spec.summarize(out))               # filtered, size-limited
    except ValidationError as e:
        return error(call.id, f"Invalid arguments: {e}")
    except PermissionDenied as e:
        return error(call.id, f"Not allowed: {e}")
    except Timeout:
        return error(call.id, "Tool timed out; try a narrower query.")

API:

POST /v1/tasks                 { prompt }                  → { task_id }
GET  /v1/tasks/{id}/events     SSE: status | tool_call | confirm_request | answer | error
POST /v1/tasks/{id}/confirm    { call_id, approved }
POST /v1/tasks/{id}/cancel

18–32 min Deep dive

You I'd go deeper on safety around side effects, keeping context and cost under control, and stopping runaway loops. Then evaluation if there's time.

Interviewer Start with side effects. What stops the agent from restarting the wrong service?

Deep dive 1 · Side effects safely

• Authorization in code: restart_service(name) checks that the requesting user is on call for the team that owns name. The model can't talk its way past it.
• Confirmation shows the exact action: "Restart checkout-api in us-east-1 (3 pods)". Never "Approve the agent's plan?"
• Idempotency keys (task ID + call ID) so a retry after a timeout doesn't restart twice.
• Dry-run first where the system supports it, and show the dry-run output in the confirmation.
• Narrow tools: restart_service with typed parameters, not run_shell(command).

Interviewer You estimated context can grow past 45,000 tokens. How do you manage that?

Deep dive 2 · Context and cost

• Filter tool outputs at the source: log search returns the top 20 matching lines with timestamps, plus counts, not 3,000 tokens of raw logs. Metrics return summary statistics and anomalies, not raw series.
• Externalize large results: store full outputs and return a handle (result_id) plus a summary; the model can call read_result(result_id, range) if it needs more.
• Compact history: after ~8 steps, replace older tool results with a running summary of findings, keeping service names, timestamps and hypotheses verbatim.
• Cache the static prefix: the system prompt and tool definitions are identical across steps and tasks, so prompt caching cuts their cost on every call.
• Parallel tool calls: independent queries (logs and metrics) run concurrently, saving wall-clock time without extra tokens.

Deep dive 3 · Stopping runaway loops

• Budgets: max 25 steps, max ~200,000 tokens, max 10 minutes, checked before every model call.
• Loop detection: the same tool with near-identical arguments three times → inject a message ("You've already run this query; summarize what you know or try a different approach"). A fourth time → stop.
• Graceful stop: on any budget hit, return a partial answer: findings so far, hypotheses, and suggested next steps. Partial is far more useful than an error.
• Cancellation: the user can cancel at any time; in-flight tool calls are cancelled if the tool supports it.

Deep dive 4 · Evaluating the agent (if time allows)

• Scenario suite: 100 past incidents replayed against recorded tool responses (logs, metrics), so runs are reproducible.
• Metrics: task success (root cause identified), steps and tokens per task, unnecessary side-effect requests, safety violations (must be zero).
• Trajectory review: read full traces of failures weekly; most fixes are better tool outputs or descriptions, not prompt tweaks.

32–37 min Trade-offs

Build first: the executor with validation, authorization and confirmations, budgets and the trace. Then tool output filtering and compaction, then the scenario suite.

37–40 min Wrap-up and likely follow-ups

Close with: "An orchestrator owns the loop, state, budgets and trace. Every tool call goes through one executor that validates arguments, authorizes against the user, asks for confirmation on side effects and returns structured errors the model can recover from. Tool outputs are filtered and history compacted to keep context and cost bounded, loops are detected, and budget hits return partial findings."

Likely follow-ups:

• When would you use multiple agents? When subtasks need separate contexts (for example parallel investigation of three services); they multiply cost and coordination failures, so justify them with evaluation.
• How do you handle a tool that's slow (minutes)? Make it asynchronous: return a job ID, let the agent do other work, and resume when the job completes.
• How would you make it safe against prompt injection in logs? Treat tool outputs as untrusted data; side effects already require confirmation, and argument provenance can raise confirmation requirements.

What interviewers listen for

• A correct loop with explicit stop conditions.
• Validation, authorization and confirmation enforced in code.
• Context growth estimated and managed.
• Runaway-loop detection and graceful partial answers.

Red flags

• while True with no budget.
• A generic shell or SQL tool given to the model.
• "The model will ask before doing anything dangerous."