Local LLM Tool Calling: Ollama and vLLM Compared

Contents

Figure 1: Local LLM Tool Calling: Ollama and vLLM Compared — the model, the template and the parser are three different things

A hosted API hides a lot of work. When you run a model yourself, you find out who makes tool calling work: the model, the chat template, the parser, or you.

I read the official Ollama and vLLM docs for this article and built a client that speaks both of their API shapes. I could not run either project on this machine, so my client ran against scripted mock servers built from the documented formats. Nothing below measures a real model.

Why Local Is Different

Hosted providers train, template and parse tool calls as one product. Locally, the pieces are separate:

1. The model has to be trained to emit tool calls at all.
2. The chat template has to render your tool definitions and past tool calls into the prompt.
3. The parser has to turn the model's raw text back into structured calls.

If any one is wrong, you get plain text where a tool call should be. Ollama bundles those pieces for you. vLLM exposes them as flags.

	Ollama	vLLM
Packaging	Pieces bundled	Pieces exposed as flags
Enabling tools	tools array on /api/chat	--enable-auto-tool-choice plus a parser
Arguments	JSON object	Depends on parser and shape
Forcing a tool	Not documented	auto, required, none, named
Best for	Laptops, experiments	Serving many requests

Ollama: The Native API

Ollama's /api/chat takes a tools array. Each entry has "type": "function" and a function object with a name, description and a JSON Schema parameters object. When the model wants a tool, the reply holds a message.tool_calls list, and each call carries a function with a name and arguments. Arguments arrive as a JSON object, not a string.

You send results back as messages with the tool role, identified by the tool's name [1]:

{"role": "tool", "tool_name": "get_temperature", "content": "22°C"}

The docs show no tool_call_id anywhere. Matching happens by tool_name, so two parallel calls to the same tool are worth testing before you rely on them. For parallel calls, the docs say to send all the calls' results back, each as its own tool message, after appending the assistant message that made the calls.

A few more details from the docs [1]:

• Streaming: gather every chunk of thinking, content and tool_calls, then send them back together with the results.
• Python helper: chat() accepts plain functions in tools and builds the schema from the docstring and type hints.
• Loop tip: the docs suggest telling the model that it is in a loop and can make multiple tool calls.
• Gaps: the docs do not list which models support tools (every example uses qwen3), do not mention a tool_choice-style control, and state no loop limit.

Ollama's model library has a "Tools" filter. On the search page I read, it showed a count of 104 out of 258 models, but the page does not explain what the tag means [3]. Treat it as a hint and test your model.

Ollama's OpenAI-Compatible Endpoint

Ollama also serves /v1/chat/completions at http://localhost:11434/v1/. Its compatibility page lists tools as supported and leaves tool_choice unchecked, which marks it as unsupported [2]. That matters if your code is written for the OpenAI client and expects to force a tool.

vLLM: Flags Decide Everything

vLLM runs an OpenAI-compatible server, and tool calling is opt-in. The docs call --enable-auto-tool-choice mandatory for auto tool choice, and --tool-call-parser picks the parser for your model family [4]. Here is the docs' example:

vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-auto-tool-choice \
  --tool-call-parser llama3_json \
  --chat-template examples/tool_chat_template_llama3.1_json.jinja

The docs list parsers by family: hermes (Hermes 2 Pro, Hermes 3, Qwen2.5), mistral, llama3_json (Llama 3.1, 3.2 and 4), granite4, deepseek_v3 and many more. Pick the wrong parser and your model's calls will not parse.

Also note the page says that without a chat template, the server cannot process chat requests: "all chat requests will error" [4].

tool_choice

vLLM supports four options [4]:

Option	Behaviour
auto	The model decides
required	Guarantees at least one call (vllm>=0.8.3)
none	No tool calls; definitions still go into the prompt unless you pass --exclude-tools-when-tool-choice-none
A named function	Force one specific tool

Named and required calls use structured outputs, which constrain the output to your schema. The docs are honest about the limit: you get "a validly-parsable function call — not a high-quality one." With auto, the output is schema-constrained only if a tool sets strict: true or the server uses --tool-strict-level. Otherwise, in the docs' words, arguments "may occasionally be malformed." Setting parallel_tool_calls to false limits a response to zero or one call.

What Goes Wrong in Practice

The vLLM docs list real caveats, and they read like a bug-report preview [4]:

• Llama 3 does not support parallel tool calls. Llama 4 does.
• Mistral 7B struggles with parallel calls.
• Smaller Llama models often fail to emit calls in the right format.
• Llama models may serialise arrays as strings.
• Pythonic parsers do not support text and tool calls in the same generation.

So a local client has to treat model output as untrusted input. Mine does.

What I Tested: A Client for Both Shapes

My client normalises both APIs into one loop. I tested it against mock servers that replay scripted replies, so this tests my code, not any model.

The two shapes differ in three ways my client handles:

1. Arguments: Ollama's native API returns an object. The OpenAI shape, which I assumed vLLM follows (I did not fetch a sample response from vLLM's docs), returns a JSON string that you must parse.
2. Matching results: Ollama uses tool_name. The OpenAI shape uses tool_call_id. My test checked each request echoed the right field.
3. Echoing the call: you send the assistant's calls back in the next request, and Ollama's arguments go back as an object while the OpenAI shape's go back as a string.

Bad output became error results, not crashes. I scripted five bad calls: truncated JSON, arguments sent as a list, a missing required argument, an unknown tool name, and an array sent as a string. For four, the client returned an instructive error message to the "model," and the scripted retry then succeeded. For the stringified array, the client silently repaired it by parsing the string, which is the failure the vLLM docs warn about. That repair is worth the five lines it costs.

Forcing a tool: my client refused to send tool_choice to the Ollama targets, because the docs I read show no such control for the native API and mark it unsupported on the compatibility page. It accepted a named function for vLLM. "Not documented" is not proof that something does not exist, so check your version.

Why Retries Matter

I ran a small calculation, with an invented failure rate, not a measured one. Take a three-call task where each call is malformed with probability p, and the client allows r retries per call. The share of tasks that finish is (1 − p^(r+1))³.

Failure rate p	No retries	One retry	Two retries
0.10	0.729	0.970	0.997
0.30	0.343	0.754	0.921

A simulation matched these figures. The math assumes each attempt fails independently. A model that repeats the same mistake after an error will do worse, so do not read this as a forecast.

The lesson is cheap to apply: return the error to the model, let it retry once or twice, then give up cleanly.

Which to Pick

• Ollama fits laptops and quick experiments. Setup is light, and the native API is simple. Check that your model actually handles tools.
• vLLM fits serving many requests, and it gives you control: parsers, templates, and forced tool choice with structured outputs.

Either way, write the loop defensively, keep your tool list short, and write detailed descriptions. Small models are less forgiving than hosted ones.

Common Pitfalls

1. Assuming a model supports tools because it is popular.
2. Forgetting --tool-call-parser, or choosing a parser for the wrong family.
3. Serving vLLM with no chat template, which errors on every chat request.
4. Sending tool_choice to Ollama and expecting it to work.
5. Matching results by position instead of tool_name or tool_call_id.
6. Trusting a stringified array without checking whether the schema wanted an array.
7. No retry and no loop limit, so one malformed call ends the task or runs forever.
8. Reading a mocked client test as a benchmark of model quality.

One Subscription Instead of Five AI Tools

Wrapping Up

Local tool calling works, but you own the seams. Ollama hides most of them and uses a simple object-and-name format. vLLM exposes them and gives you flags to force and constrain calls. In both, treat model output as untrusted: parse carefully, repair what is safely repairable, and return clear errors.

Start with one tool, one model and one loop. Test the failure cases on purpose before you trust it. Then add a second tool and measure again.

If you are wiring these into a larger system, the MCP client walkthrough covers the loop around the model, and how LLMs decide to call a tool explains the decision the parser is trying to recover.

Frequently Asked Questions

Does Ollama support tool calling?

Yes. The native /api/chat endpoint takes a tools array of function definitions with a JSON Schema parameters object, and a reply that wants a tool carries message.tool_calls. You send results back as messages with the tool role, matched by tool_name. The OpenAI-compatible endpoint at /v1 also lists tools as supported, but the compatibility page leaves tool_choice unchecked, so forcing a specific tool is not available there.

Why does my local model return text instead of a tool call?

Three separate things have to line up: the model must be trained to emit calls, the chat template must render your tool definitions and prior calls into the prompt, and the parser must turn the raw text back into structured calls. Hosted providers ship all three as one product; locally they are separate. With Ollama most of it is bundled, so the usual culprit is a model that does not really support tools. With vLLM the usual culprits are a missing parser flag or a chat template that does not exist.

What does --tool-call-parser do in vLLM?

It tells the server which format to expect back from the model family, and the vLLM docs call it mandatory alongside --enable-auto-tool-choice for automatic tool choice. The parsers are listed per family: hermes for Hermes and Qwen2.5, mistral, llama3_json for Llama 3.1, 3.2 and 4, granite4, deepseek_v3 and others. Pick the wrong one and the model's calls will not parse, which looks like the model ignoring your tools.

Can I force a specific tool on a local model?

On vLLM, yes. It supports auto, required, none and a named function, and the named and required modes use structured outputs that constrain the output to your schema. The docs are explicit that this yields a validly parsable call rather than a high-quality one. With Ollama's native API the docs show no tool_choice-style control, and the compatibility page marks it unsupported, so a client written for the OpenAI SDK should not assume it can force a tool.

How many retries should a tool loop allow?

One or two, then give up cleanly. If each call fails with probability p and you allow r retries, a three-call task completes with probability (1 − p^(r+1))³. With p of 0.10 that is 0.729 with no retries, 0.970 with one and 0.997 with two. The numbers assume independent failures, and a model that repeats the same mistake will do worse, so treat this as a reason to retry once rather than a forecast.

Why do Llama models fail parallel tool calls?

Because support varies by generation and by template. The vLLM docs state that Llama 3 does not support parallel tool calls while Llama 4 does, and that Mistral 7B struggles with them. They also note that smaller Llama models often fail to emit calls in the right format and that Llama models may serialise arrays as strings. Setting parallel_tool_calls to false limits a response to zero or one call, which is the safer default for small models.

Sources

• [1] Ollama, Tool calling — the tools array shape, message.tool_calls, tool role messages, streaming chunks, the Python helper and the loop tip: docs.ollama.com/capabilities/tool-calling
• [2] Ollama, OpenAI compatibility — /v1/chat/completions, tools marked supported, tool_choice unchecked: docs.ollama.com/api/openai-compatibility
• [3] Ollama, model library with the Tools filter — the 104-of-258 count with no published explanation of the tag: ollama.com/search?c=tools
• [4] vLLM, Tool calling — --enable-auto-tool-choice, the per-family parsers, tool_choice modes, structured outputs and the caveat list: docs.vllm.ai/features/tool_calling
• [5] vLLM, Online serving — OpenAI-compatible server flags and chat-template handling: docs.vllm.ai/serving/openai_compatible_server

Related Articles

• Build an MCP Client in Python: Connect Any LLM to MCP Tools
• How LLMs Decide to Call a Tool: A Step-by-Step Walkthrough
• Function Calling vs Tool Calling: What's the Difference?
• MCP vs Function Calling: Difference and When to Use Each