Home / Coding with AI APIs

Tool use, streaming and agents

How function calling works, the agent loop, streaming responses, and when an agent is the wrong choice.

11 min read

Tool use (function calling)

You describe tools to the model with a name, description and JSON schema for arguments. The model can respond with a tool call instead of text; your code executes it and sends the result back.

tools = [{
    "name": "get_order_status",
    "description": "Look up the status of a customer's order by order id.",
    "input_schema": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"],
    },
}]

messages = [{"role": "user", "content": "Where is order A1234?"}]

while True:
    response = client.messages.create(model=MODEL, max_tokens=1024,
                                      tools=tools, messages=messages)
    messages.append({"role": "assistant", "content": response.content})

    if response.stop_reason != "tool_use":
        break  # final answer

    results = []
    for block in response.content:
        if block.type == "tool_use":
            output = run_tool(block.name, block.input)   # validate + authorize!
            results.append({"type": "tool_result",
                            "tool_use_id": block.id,
                            "content": output})
    messages.append({"role": "user", "content": results})

Key points interviewers look for:

  • The model never executes anything. Your code does, so your code must validate arguments and check permissions.
  • Tool descriptions are prompts: clear names and descriptions drastically improve tool choice.
  • Handle multiple and parallel tool calls in one turn.
  • Return errors as tool results so the model can recover, rather than crashing the loop.
  • Put a limit on iterations and total tokens to stop runaway loops.

The agent loop

An agent is a model in a loop: observe → decide → act with a tool → observe the result → repeat until done.

Common patterns, from simplest to most autonomous:

  1. Single call with good context.
  2. Prompt chaining: fixed sequence of steps.
  3. Routing: classify, then send to a specialized prompt.
  4. Parallelization: run subtasks concurrently and aggregate.
  5. Orchestrator–workers: one model plans and delegates to others.
  6. Evaluator–optimizer: one model generates, another critiques, loop.
  7. Autonomous agent: open-ended loop with tools.

Start with the simplest pattern that works. Autonomous agents are harder to test, slower, more expensive and riskier. Use them when the number of steps genuinely can't be predicted in advance.

Model Context Protocol (MCP)

MCP is an open protocol for connecting models to tools and data sources through a standard client–server interface, so a tool integration can be written once and used by many AI applications. Know what it is at a high level and the security implications of connecting third-party servers.

Streaming

Streaming APIs return server-sent events: incremental text deltas, tool-call argument fragments, and a final message with usage and stop reason.

  • Render text deltas as they arrive for a fast perceived response.
  • Buffer tool-call arguments until complete before parsing JSON.
  • Handle disconnects and partial output; record usage from the final event.
  • If you moderate output, decide whether to check chunks incrementally or hold back until a sentence boundary.

Handling API errors

  • 429 rate limit / 529 or 503 overloaded: retry with exponential backoff and jitter; honour Retry-After.
  • 400 bad request: don't retry blindly; usually a schema or context-length bug.
  • Timeouts: use streaming for long outputs and set generous but bounded timeouts.
  • Track tokens to stay inside the context window: trim or summarize old conversation turns.