How does function calling / tool use work under the hood?

30-second answer

You send tool definitions (name, description, JSON schema). The model is trained to emit a structured tool call when appropriate. Your application parses it, executes the function, and returns the result as a tool-result message. The model then continues, possibly calling more tools, until it produces a final answer. The model never runs code itself.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask whether they want the API-level flow, what happens inside the model, or both. Framing: "Tool use is a protocol between your code and the model: you describe tools, the model emits structured requests, your code executes them and returns results. I'll walk through the message flow, how the model is trained to do this, and the engineering responsibilities on your side."

5–15 min Core explanation: the message flow

Define tools with a name, a description and a JSON Schema for arguments.
Send the conversation plus tool definitions to the model.
The model either answers in text or returns one or more tool-use blocks: tool name, an ID, and JSON arguments. The response's stop reason indicates a tool call.
Your code validates the arguments, checks permissions, executes the function.
Return results as tool-result messages referencing the tool-use ID.
The model continues: it may call more tools or produce the final answer.

user: "What's the weather in Redmond and should I bike?"
assistant → tool_use: get_weather({"city": "Redmond"})
app       → tool_result: {"temp_c": 9, "rain_mm": 4}
assistant → "It's 9°C with rain expected; consider the bus."

15–25 min Deep dive: inside the model

Tool definitions are serialized into the prompt (often in a system section) in a format the model saw during training.
Models are fine-tuned on many examples of deciding when to call tools and producing well-formed arguments, followed by tool results and continuations.
The model generates the tool call as tokens, like any output; providers parse it into structured blocks. Some use constrained decoding to guarantee schema-valid arguments.
Tool choice settings let you force a specific tool, require some tool, or let the model decide.
Parallel tool calls: the model can request several independent calls in one turn; run them concurrently.

Writing good tool definitions:

Clear, specific names (search_orders_by_customer, not query).
Descriptions that say when to use the tool, what it returns, and limitations.
Tight schemas: enums, required fields, formats.
Fewer, well-designed tools beat many overlapping ones.

25–32 min Engineering responsibilities

Validation and authorization: the model's arguments are untrusted input. Check types, ranges and that the user may act on that resource.
Errors as results: return structured error messages so the model can recover.
Side effects: idempotency keys, confirmations for destructive actions.
Result size: return relevant fields only; large results bloat context and cost.
Timeouts and limits on tool execution and iterations.

32–37 min Production example

A sales assistant had a query_crm tool taking free-form SQL. The model sometimes queried other teams' accounts and produced slow queries. Replacing it with three narrow tools (get_account, list_open_deals, get_contact) with typed parameters and server-side scoping to the user's territory fixed both security and reliability, and tool selection accuracy improved.

37–40 min Likely follow-ups

Does the provider run my tools? For your custom tools, no; some providers offer built-in server-side tools (like web search or code execution) that they run.
How do you test tool use? Eval cases checking the right tool and arguments are chosen; mocked tool results for deterministic tests.

What interviewers listen for

Accurate message flow including tool-use IDs and results.
The model requests, your code executes and must validate.
Tool design quality.

Red flags

"The model calls the API directly."