LiteLLM Local Models: Ollama vs vLLM and the /v1 Trap

Contents

Figure 1: LiteLLM Local Models: Ollama vs vLLM and the /v1 Trap — the model runs fine; the base URL is what breaks

Running a model on your own machine is the easy part. Making every tool talk to it through one stable name is where people get stuck. LiteLLM helps, but its docs for Ollama and vLLM disagree with each other in one place that will cost you an afternoon if you miss it.

I ran neither Ollama, vLLM nor LiteLLM, since none are installed here. The settings below come from the LiteLLM and vLLM docs pages. My script lints configs against those quoted rules and probes URLs on mock servers.

Ollama: Pick the Right Prefix

For Ollama, LiteLLM has two prefixes, and the docs are clear about which one they like. Most examples use ollama/, but the page says: "We recommend using ollama_chat for better responses" [1].

Prefix	What it calls	Use it for
ollama_chat/	Ollama's POST /api/chat	The recommended chat path
ollama/	Ollama's /api/generate	FIM-style calls sent to /v1/completions

The ollama_chat/ prefix sends requests to Ollama's POST /api/chat. The ollama/ prefix is documented for FIM-style calls on /v1/completions, which LiteLLM routes to Ollama's /api/generate [1].

A proxy entry using the recommended prefix, built from the docs' examples [1]:

model_list:
  - model_name: local-llama
    litellm_params:
      model: ollama_chat/llama3.1
      api_base: http://localhost:11434
      keep_alive: "8m"  # optional; use -1 for forever
    model_info:
      supports_function_calling: true

Every example on the page passes api_base="http://localhost:11434" explicitly. The page states no default and does not mention an OLLAMA_API_BASE variable, so set api_base yourself [1]. keep_alive is optional and overrides the server default, with -1 meaning forever, and format="json" turns on JSON mode [1].

Worth knowing: the page's own headline claim is "LiteLLM supports all models from Ollama", with no list behind it [1]. Treat it as "the route exists", not "every model behaves the same."

Ollama and Tool Calling

Here is a caveat worth reading twice. The page says "not all ollama models support function calling," and that "litellm defaults to json mode tool calls if native tool calling not supported" [1]. You can also call litellm.register_model(...) with supports_function_calling, or set the same flag under model_info [1].

So if your coding agent calls tools and you get odd results, check the model first, not the proxy. JSON-mode tool calls are valid JSON, not real invocations — a distinction that looks fine in logs and breaks an agent loop.

vLLM: Use hosted_vllm/

LiteLLM's vLLM page says the route is hosted_vllm/ for an OpenAI-compatible server. It marks vllm/ as "[DEPRECATED] for vLLM sdk usage" [2]. The proxy entry from the docs [2]:

model_list:
  - model_name: gpt-oss-120b
    litellm_params:
      model: hosted_vllm/gpt-oss-120b
      api_base: https://hosted-vllm-api.co

You can set HOSTED_VLLM_API_BASE as an environment variable instead. An API key is optional: the page shows HOSTED_VLLM_API_KEY with the comment "if your VLLM server requires an API key" [2].

The page also lists the endpoints it routes: /chat/completions, /embeddings, /completions, /rerank and /audio/transcriptions [2]. That is a useful sanity check — if you are trying to reach an endpoint not on that list, the prefix is not the problem.

Two gaps remain: the page says LiteLLM "supports all models on VLLM" but gives no list backing it up, and it says nothing about tool calling [2].

The /v1 Disagreement

Now the trap. vLLM's own LiteLLM page uses http://{your-vllm-server-host}:{your-vllm-server-port}/v1 as the api_base, for chat and embeddings alike [3]. LiteLLM's vLLM page uses a base without /v1 and never explains the difference [2]. LiteLLM's separate OpenAI-compatible page says that if you get a "Not Found Error," your api_base should end in /v1 [4].

My advice is to start with the /v1 form for vLLM, and drop it if requests fail. Either way, test it.

My checker flagged hosted_vllm entries without /v1 as a hint, not an error, because the docs do not settle it. Its probe shows the basic logic: against a mock vLLM-style server, the bare base answered only when the client added /v1 itself, and a base ending in /v1 answered at /models. A 404 means a wrong base or path, while no answer at all means nothing is listening. Those two have different fixes.

Could You Use the openai/ Prefix Instead?

Per LiteLLM's OpenAI-compatible page, yes. Its config example uses openai/<model> with an endpoint whose api_base ends in /v1 [4]. The vLLM page does not mention openai/, so choose one and stay consistent. I would use hosted_vllm/ for vLLM because both vendor pages show it [2][3].

Reaching the Local Model From Your Tools

Once the proxy is up, clients ask for your model_name (like local-llama) at the proxy address. The docs' examples run the proxy on http://0.0.0.0:4000 with a bearer token [1][2]. Three practical points:

1. Keep the proxy and the model server on the same network path. If LiteLLM runs in Docker, localhost means the container. I did not test containers, and the pages I read do not cover this, so check your own networking.
2. Give the proxy a virtual key and do not expose Ollama or vLLM directly.
3. Pair local with hosted. Use a local model first and a hosted one as a fallback, which the earlier retries and fallbacks setup supports [5].

Quick Troubleshooting

Symptom	First thing to check
"Not Found"	Try the other api_base spelling, with or without /v1
Connection refused	Nothing is listening at that address; check the server and port
Odd tool-call output	Whether your Ollama model supports native tool calling
Slow or empty responses	The model server's own logs; the proxy only forwards
Wrong prefix	ollama/ for completions-style, ollama_chat/ for chat, hosted_vllm/ for vLLM

Common Pitfalls

1. Copying an ollama/ example and wondering why chat quality dropped. The page recommends ollama_chat/.
2. Assuming api_base has a default for Ollama. Every documented example sets it.
3. Setting supports_function_calling: true on a model that cannot call tools. The flag describes the model; it does not add the ability.
4. Mixing vllm/ into a new config. It is marked deprecated for SDK usage.
5. Guessing the /v1 suffix instead of testing both. Two docs, two spellings, one working answer for your server.
6. Sending embeddings to a vLLM route that expects a chat model. The prefix is shared; the endpoint list is not.
7. Exposing Ollama on 0.0.0.0:11434 because the proxy needed it. Bind it to the proxy's network instead.
8. Skipping the fallback until the local model is in the request path. That is the day it will be cold or offline.

Wrapping Up

Routing LiteLLM to local models comes down to three choices: the right prefix, the right base URL and honest expectations about tool calling. Use ollama_chat/ and hosted_vllm/, keep the /v1 question testable and write the working spelling down.

Try one model end to end before you add a fallback chain.

If the gateway is new to you, the LiteLLM proxy guide covers the quickstart, the virtual keys guide keeps the local endpoint off your team's raw access, and the load balancing guide explains what happens when a local and a hosted deployment share one model name.

Frequently Asked Questions

Should I use ollama or ollama_chat in LiteLLM?

Use ollama_chat. LiteLLM's Ollama page says "We recommend using ollama_chat for better responses" because that prefix sends requests to Ollama's POST /api/chat. The plain ollama/ prefix is documented for FIM-style calls on /v1/completions, which LiteLLM routes to Ollama's /api/generate endpoint.

Does LiteLLM need api_base set for Ollama?

Yes. Every example on LiteLLM's Ollama page passes api_base="http://localhost:11434" explicitly. The page states no default and does not mention an OLLAMA_API_BASE environment variable, so set api_base in the config or the call rather than assuming a default is applied.

Should the vLLM api_base end in /v1?

Start with /v1. vLLM's own LiteLLM integration page uses http://{your-vllm-server-host}:{your-vllm-server-port}/v1 as api_base for both chat and embeddings, while LiteLLM's vLLM page shows a base without /v1. LiteLLM's OpenAI-compatible page separately says a Not Found Error means the api_base should end in /v1. Test whichever you choose and record the spelling that works.

Is vllm/ or hosted_vllm/ the LiteLLM prefix for vLLM?

hosted_vllm/. LiteLLM's vLLM page lists hosted_vllm/ for an OpenAI-compatible server and marks vllm/ as deprecated for vLLM SDK usage. Both vLLM's own integration page and LiteLLM's examples use hosted_vllm/, so stay with it and keep the model id after the prefix.

Why do tool calls fail on a local Ollama model through LiteLLM?

Because not every Ollama model supports function calling. LiteLLM's page states that not all ollama models support function calling and that litellm defaults to JSON-mode tool calls when native tool calling is unsupported, which produces valid JSON rather than real tool invocations. You can declare support with litellm.register_model or model_info.supports_function_calling, but the model itself still has to be capable.

Can I mix a local model with a hosted fallback in LiteLLM?

Yes, and you should. A local deployment can be slow, cold or offline, so placing a hosted deployment behind the same model group as a fallback turns an outage into a slower response. Configure it with router_settings fallbacks or per-model fallbacks, and keep the local model first if cost is the reason you are self-hosting.

Sources

• [1] LiteLLM, Ollama — the "We recommend using ollama_chat for better responses" note, ollama_chat/ sending requests to POST /api/chat, ollama/ for FIM-style calls on /v1/completions routed to /api/generate, the api_base="http://localhost:11434" in every example, the optional keep_alive with -1 for forever, format="json" for JSON mode, the "not all ollama models support function calling" caveat with JSON-mode fallback, litellm.register_model and supports_function_calling, and the "supports all models from Ollama" headline: docs.litellm.ai/docs/providers/ollama
• [2] LiteLLM, vLLM — hosted_vllm/ for an OpenAI-compatible server, vllm/ marked "[DEPRECATED] for vLLM sdk usage", the hosted_vllm/gpt-oss-120b proxy config with a base that has no /v1, HOSTED_VLLM_API_BASE, the optional HOSTED_VLLM_API_KEY, the routed endpoint list including /chat/completions, /embeddings, /completions, /rerank and /audio/transcriptions, and the "supports all models on VLLM" claim with no list: docs.litellm.ai/docs/providers/vllm
• [3] vLLM, LiteLLM integration — the hosted_vllm/ prefix for chat and embeddings, and the api_base of http://{your-vllm-server-host}:{your-vllm-server-port}/v1 used for both: docs.vllm.ai/en/latest/deployment/frameworks/litellm
• [4] LiteLLM, OpenAI-Compatible Endpoints — the openai/ prefix for OpenAI-compatible providers, the "If you see Not Found Error when testing make sure your api_base has the /v1 postfix" rule, and the http://vllm-endpoint.xyz/v1 example: docs.litellm.ai/docs/providers/openai_compatible
• [5] LiteLLM, Reliability and fallbacks — fallback configuration so a local deployment can hand off to a hosted one, alongside retries, cooldowns and timeouts: docs.litellm.ai/docs/proxy/reliability

Related Articles

• Aider, Claude Code, Cursor: One LiteLLM Proxy for All
• LiteLLM Retries and Fallbacks: Stop Retrying a 401
• LiteLLM Load Balancing: Your RPM Limit Is a Suggestion