How LLMs Decide to Call a Tool: A Step-by-Step Walkthrough

Contents

Figure 1: How LLMs Decide to Call a Tool: A Step-by-Step Walkthrough — the model writes a request; your code does the touching

When an AI assistant looks up the weather, it did not reach out and touch a server. It wrote a request, and your code did the touching. That one fact clears up most of the confusion about tool calling.

I read Anthropic's tool-use docs and OpenAI's function-calling guide for this walkthrough, then built a small demo to test the mechanics. Wherever I am reporting a demo result rather than a documented rule, I say so — some of my numbers come from a crude proxy, not a real model, and I flag those explicitly.

Step 1: You Describe the Tools

The model never sees your code. Anthropic's docs say "Claude never sees your implementation; it only sees the schema you provided and the result you returned." Each tool in the tools array has three parts:

Part	What it does
Name	The identifier the model emits back to you
Description	The text the model reads when deciding whether to call it
Input schema (JSON Schema)	The parameters, their types and which are required

OpenAI's guide describes the same shape, with a parameters schema and an optional strict flag. The tool list travels with every request, which is why it is a cost you pay per call rather than once.

Anthropic's token table shows the API even adds its own system prompt when you pass tools — 286 tokens on Opus 5.5 with the default setting, per the page. You are billed for scaffolding you did not write.

Step 2: The Model Decides, From Text Alone

This is the step everyone asks about, and the docs are modest about it. Anthropic says "Claude determines when to call a tool based on the user's request and the tool's description." With the default auto setting, "it calls a tool when the request maps to that tool's described capability and the answer isn't already in context."

Notice what is missing: no scoring formula, no router. The model reads the request and the descriptions as ordinary text and decides, using behaviour it learned in training. Anthropic's page adds a warning: this behaviour "is not guaranteed, especially for more ambiguous prompts and for less capable models." OpenAI's guide is similarly plain: the model "may decide that it needs data or functionality provided by a tool."

So your descriptions are the only steering wheel you get. More on that below.

Step 3: The Model Emits a Request, Not a Result

If the model decides to call a tool, it returns a structured block. On Anthropic's API that is a tool_use block with an id, the tool name, and an input object that follows your schema. The response's stop_reason becomes "tool_use". On OpenAI's side the model returns a tool call with a call_id.

Both providers say a single response can hold several calls. OpenAI's guide advises you to assume there will be several. The model proposes, and nothing has run yet.

{
  "stop_reason": "tool_use",
  "content": [
    {"type": "text", "text": "Let me check that."},
    {"type": "tool_use", "id": "toolu_01A", "name": "get_weather",
     "input": {"city": "Paris"}}
  ]
}

Step 4: Your Code Does the Work

Your application reads each request, runs the real operation, and catches whatever goes wrong. Anthropic calls this "a contract between your application and the model": the model "never executes anything on its own."

Server tools like web search are the exception, because Anthropic runs those for you. That distinction decides where your validation, retries and rate limits live: everything client-side is your responsibility.

Step 5: You Send the Result Back

You append the assistant's turn, then a user turn containing a tool_result for each call. Anthropic's docs spell out the ordering rules, and they are strict:

1. Each result must reference the matching tool_use id.
2. Results must immediately follow the tool-use message. No messages in between.
3. In the user message, the tool_result blocks come first. Any text goes after them.

I wrote a small checker from these rules and fed it four broken transcripts:

Transcript defect	Caught
Text before a result	Yes
A missing result	Yes
A wrong id	Yes
An extra message in between	Yes
Well-formed turn with two parallel calls	Passed

Break these rules and the real API returns a 400 error, according to the docs. That is a cheap failure to test for, and a checker like this catches it before a request is ever sent.

Step 6: The Loop

Then the model reads your results and either answers or asks for more tools. Anthropic's page gives the canonical loop: repeat while stop_reason is "tool_use", and exit on any other reason — it lists end_turn, max_tokens, stop_sequence and refusal.

I drove that loop with a scripted fake model that asks for tools twice and then answers. It took three model calls and six messages.

One gap worth knowing about: the page I read states no cap on loop iterations. I added my own limit to the demo, and a model that never stops hit it. Add yours too, along with a token budget, because the failure mode is a bill rather than an error.

What Actually Shapes the Decision

Since the decision comes from reading text, text quality matters most. Anthropic's tool-definition page says it bluntly: "Provide extremely detailed descriptions. This is by far the most important factor in tool performance." It asks for what the tool does, when to use it "(and when it shouldn't)", what each parameter means, and caveats. Its rule of thumb is at least three or four sentences per tool.

Other levers the docs name:

• Consolidate. "Fewer, more capable tools reduce selection ambiguity." One tool with an action parameter beats three near-twins.
• Namespace. Prefix names by service, like github_list_prs.
• Keep the list short. OpenAI suggests fewer than 20 functions at the start of a turn, and calls that a soft suggestion. Anthropic's page names a tool search feature for very large sets.
• Steer with the system prompt. Anthropic's example nudges: "Use the tools to investigate before responding" for more calls, or "Use your judgment" for fewer.
• Use tool_choice. On Anthropic's API, auto lets the model choose, any forces some tool, tool forces one named tool, and none blocks tools.

Forcing has side effects. With any or tool, the API prefills the reply, so the model emits no explanation before the call, and a forced call skips thinking. The page also says any and tool return a 400 error on Claude Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1, and it recommends auto with strict tool use instead. Its code sample uses a model name that differs from that table, so check the page for your exact model. Changing tool_choice also invalidates cached message blocks.

What the Demo Says (and Doesn't)

I could not call a real model from here, so I built a crude proxy: score each tool description against a request using TF-IDF word matching. A real model reads far better, so do not treat these numbers as forecasts. They only show how much information a description gives you.

Setup	Proxy score	How to read it
One-line descriptions ("Gets weather.")	21% of 24 requests matched correctly	Much of the gap is the proxy missing words like "rain"
Detailed what/when/not descriptions	83%	Read as "more to go on", not as LLM behaviour
Plus two near-duplicate tools	71%	Counts picking a duplicate as wrong, so it overstates the harm
Near-ties among top two	3 → 4 of 6 requests	A tiny sample; weak evidence that duplicates blur the choice

Any "call a tool or just answer" rule makes two mistakes: skipping needed calls and making needless ones. In my threshold table, the lowest cutoff skipped 2 of 24 needed calls and made 1 needless call out of 8. The highest skipped 23 of 24.

Tool definitions are not free. By my rough estimate (characters divided by four), 8 tools cost about 864 tokens and 80 tools about 8,665, and every request carries them.

In my view, the biggest takeaway is dull but useful: the model can only choose well between tools it can tell apart.

When Things Go Wrong

Tools fail, and the loop must survive it. Anthropic's docs say to return the failure in the result with is_error: true, and to "write instructive error messages." Instead of "failed", say what went wrong and what to try next. For invalid requests, the page says Claude retries "2-3 times with corrections before apologizing." In my scripted loop, a failing tool returned its error to the model and the transcript still passed the checker.

The docs add a security point: tool results often carry content from outside your control. Keep it inside tool_result blocks, not your system prompt, because text in a result is data, while text in the prompt is instruction.

Common Pitfalls

1. Assuming the model ran the tool. It wrote a request and stopped.
2. Sending results out of order. The id match and immediate adjacency are enforced, not advisory.
3. Leaving the loop uncapped. Without a cap, a confused model keeps asking and the bill keeps growing.
4. Writing one-line descriptions. This is the highest-leverage line of text in the whole integration.
5. Shipping three near-identical tools. The model has to guess, and its guess is not reproducible.
6. Using tool_choice: any as a reliability fix. It breaks on several current models and skips the model's reasoning.
7. Treating retrieved text as instructions. Injection arrives through tool results more often than through user prompts.

Quick-Start Checklist

[ ] every tool has a name, a description and a JSON Schema
[ ] descriptions say what it does, when to use it AND when not to
[ ] near-duplicate tools merged or namespaced
[ ] tool list under ~20, with tool search if it must be larger
[ ] tool_choice left on auto unless forcing is genuinely needed
[ ] argument validation before execution
[ ] tool_result id matches, adjacency respected, blocks before text
[ ] is_error set with an actionable message
[ ] iteration cap and token budget configured
[ ] retrieved content kept inside tool_result, never in the system prompt

One Subscription Instead of Five AI Tools

None of the mechanics above change because of a subscription, but the bill you are optimising against does.

Frequently Asked Questions

Does the model execute the tool itself?

No. The model writes a structured request containing a tool name and arguments, then stops and waits. Your application runs the real operation, catches failures and returns the result. Anthropic's docs put it plainly: Claude never sees your implementation, only the schema you provided and the result you returned. The exception is server-side tools such as web search, where the provider runs the tool for you.

How does an LLM know which tool to call?

It reads your request and your tool descriptions as ordinary text and decides using behaviour learned in training. There is no scoring formula or router. Anthropic says it calls a tool when the request maps to that tool's described capability and the answer is not already in context, and warns that this behaviour is not guaranteed, especially for ambiguous prompts and less capable models. Your descriptions are the only steering wheel you get.

What does stop_reason tool_use mean?

On Anthropic's API it means the model ended its turn by asking for a tool rather than by producing a final answer. The response contains a tool_use block with an id, the tool name and an input object, and nothing has run yet. Your code executes it, returns a tool_result, and repeats while stop_reason stays tool_use, exiting on end_turn, max_tokens, stop_sequence or refusal.

Do tool results have to come immediately after the tool call?

Yes. Each tool_result must reference the matching tool_use id, results must immediately follow the message containing the tool-use block with no messages in between, and inside that user message the tool_result blocks come first with any text after them. Break those rules and the API returns a 400 error. I wrote a checker from the documented rules and it flagged all four broken transcripts I fed it.

Is there a limit on how many times a tool loop can run?

Not one stated on the page I read. The documented loop simply repeats while stop_reason is tool_use and exits on any other reason, so a model that never stops asking can keep going. Add your own iteration cap and a max-token budget. In my demo, a scripted model that never stopped ran until my own limit fired, which is exactly the failure mode a cap exists for.

How many tools should I give the model in one request?

Fewer than you think. OpenAI suggests fewer than 20 functions at the start of a turn and calls that a soft suggestion; Anthropic offers a tool search feature for very large sets. Tool definitions also cost tokens on every request: by my rough estimate, 8 tools cost about 864 tokens and 80 tools about 8,665. Fewer, better-described tools improve selection accuracy and cut cost at the same time.

Sources

• Anthropic: tool use overview — tool_use blocks, stop_reason, tool_choice options, the canonical loop and the loop's stopping conditions.
• Anthropic: implementing tool use — tool definitions, the description guidance, is_error handling and the result-ordering rules.
• OpenAI: function calling guide — the tools and parameters shape, the optional strict flag, call_id and the advice about multiple calls per response.

Numbers in "What the Demo Says" come from my own TF-IDF proxy and checker, not from either vendor. The token estimates there are character counts divided by four, which is a rough approximation rather than a measurement of the API's tokeniser.

Model lists, token counts and tool_choice behaviour change between releases, so verify against the page for your exact model.

Wrapping Up

A tool call is a conversation with a gap in the middle. The model reads your descriptions, writes a structured request, and waits. Your code runs the work, returns the result in the right order, and the loop continues until the model stops asking.

You cannot see inside the decision, but you control its inputs: write detailed descriptions, remove duplicates, keep the list short, and return errors the model can act on. Pick your three most confusing tools this week and rewrite their descriptions to say when to use them and when not.

Which of your tool descriptions would survive being read by someone who has never seen your code? That reader is the model. The function calling versus tool calling breakdown covers the vocabulary, and the agent build guide covers what to do once the loop runs.

Related Articles

• Function Calling vs Tool Calling: Are They the Same?
• Function Calling and Tool Use: How LLMs Take Actions
• MCP vs Function Calling: Difference and When to Use Each
• Build Your First AI Agent in Python
• AI Agent Guardrails: Preventing Runaway Actions