LiteLLM Tutorial: One API for Every LLM Provider

Contents

Figure 1: One request shape to rule the adapter collection

What LiteLLM does, how calls, routing and cost tracking work, and the honest untested parts

Every LLM provider has its own SDK, its own request shape and its own error messages. Wire up three of them and your codebase starts to look like a travel adapter collection. Can one call work for all of them?

That's the pitch behind LiteLLM. This tutorial covers what it does, how the basic call works, how the Router handles retries and fallbacks, and when you'd want the proxy instead — and marks what couldn't be tested, because the library couldn't be installed in this sandbox. The gateway role it plays is exactly the one the LLMOps production guide describes for model routing and failover.

What LiteLLM Is

LiteLLM's docs describe it as an open-source library that gives you "a single, unified interface to call 100+ LLMs," using the OpenAI format. It comes in two forms:

• The Python SDK. You import it into your app and call models directly. It includes a Router for retries, fallbacks and load balancing.
• The proxy (LLM gateway). A self-hosted, OpenAI-compatible server for teams. The docs say it adds virtual keys, budgets, centralized logging, guardrails, caching and an admin UI.

Pick the SDK when one app needs to talk to several models. Pick the proxy when many apps or teams share access and someone needs to track spend.

The Basic Call

The docs' example uses completion() with a provider-prefixed model name:

import os
from litellm import completion

os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
    model="openai/<model-name>",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
)
print(response.choices[0].message.content)

The docs' model name is replaced with a placeholder here, since model names change quickly. Put in whichever current model you use.

The key claim from the docs: "Every response follows the OpenAI Chat Completions format, regardless of provider." That means response.choices[0].message.content works whether the model came from OpenAI, Bedrock or somewhere else — the same normalization the LLM API comparison wishes every wrapper delivered.

To switch providers, change the model string and the matching credentials. The fetched docs show the prefix style with openai/... and bedrock/... examples; there's no full naming guide on the providers page, so check the page for your provider.

FYI: none of this code ran here. More on that in the notes.

Why Bother with a Unified Interface?

Fair question. Here's the honest list.

What you gain:

• One response shape. Your parsing code stays the same.
• Easy model swaps. Comparing two providers becomes a one-line change.
• Consistent errors. The docs say LiteLLM "maps exceptions across all providers to their OpenAI counterparts," so one except block can cover several providers.
• Cost tracking and callbacks built in.

What you give up:

• A dependency. You're now trusting LiteLLM to keep up with each provider — the provider-update hazard the LLMOps vs MLOps comparison flags as a first-class risk.
• Provider-specific features. A unified interface can lag behind new or unusual features. That wasn't tested here, so check before you rely on one.
• One more thing to debug. When a call fails, is it the provider or the wrapper?

IMO, the trade is worth it once you use two or more providers. For a single provider, the official SDK is simpler.

Streaming

The docs list streaming as a feature: pass stream=True to receive chunks as they're generated. The chunk format wasn't fetched, so no loop is shown here — read the streaming page before writing one.

Error Handling

The exception-mapping page is thin in the version fetched. It says LiteLLM maps provider exceptions to their OpenAI counterparts, and its import example names BadRequestError. The table of exception classes and status codes was empty.

So here's the safe advice: catch the specific classes the docs list, and log the original message too. Don't build retry logic on class names you haven't confirmed.

Retries, Fallbacks and Load Balancing with the Router

This is the feature that justifies the library for many teams. The Router takes a model_list. Each entry pairs a model_name alias with litellm_params (provider model, credentials, endpoint). Entries that share an alias are load-balanced together.

A minimal shape, based on the docs' description (untested):

from litellm import Router

router = Router(model_list=[
    {"model_name": "chat", "litellm_params": {"model": "openai/<model-a>", "api_key": "..."}},
    {"model_name": "chat", "litellm_params": {"model": "bedrock/<model-b>"}},
])

Your app asks for "chat", and the Router picks a deployment.

Routing Strategies

Strategy	What it does	Doc note
Default (simple-shuffle)	Picks randomly; a weight makes some deployments more likely	"For best performance in production"
Usage-based	Tracks TPM and RPM per deployment	"Not recommended for production due to performance impacts"
Latency-based	Prefers deployments with recent fast responses	Uses recent response times
Cost-based	Prefers the cheapest deployment	Good for budget-driven traffic

Priority and Fallbacks

Set order to rank deployments ("lower values = higher priority"). If one fails, the router moves to the next level, and each level gets its own retries. If every level is exhausted, it falls through to any configured fallbacks.

Retries

Per the docs, rate-limit errors get exponential backoff, and generic errors retry immediately. num_retries can be set in four places, and the request header wins, then the request body, then the deployment's params, then global settings.

Figure 2: When one provider throttles, the Router decides what your users see next

Cooldowns

A failing deployment can be sidelined. Defaults per the docs are allowed_fails: 3 and cooldown_time: 5s. Triggers include an immediate cooldown on a 429, a failure rate above 50% in the current minute, and non-retryable errors such as 401, 404 and 408. The docs are inconsistent on duration — one passage says "a minute," while the defaults say 5 seconds. Check the real value in your config and test it.

Tracking Cost

LiteLLM returns token usage by default, and the docs say it returns a response_cost on every call:

print(response._hidden_params["response_cost"])

There are two helpers. completion_cost(completion_response=response) returns the total USD cost as a float. cost_per_token(...) returns prompt and completion costs separately, using a price list from LiteLLM's own site. That list is third-party data, so treat the numbers as estimates and compare them with your provider's bill — a habit the cost section of the observability guide pushes for every layer of the stack.

When to Use the Proxy

The proxy is a self-hosted, OpenAI-compatible gateway. The docs say any client that works with OpenAI works with it, "with no code changes." That makes it handy when:

• Several teams share provider keys, and you want virtual keys and budgets.
• Non-Python services need access.
• You want central logging and guardrails.

The cost is another service to run, secure and monitor. The proxy's config reference couldn't be read in the fetched version (empty tables), so no config file is shown here — use the official quick start for that part.

A Sensible Adoption Path

1. Start with completion() in one service. Swap the model string, and check your outputs.
2. Add the Router once you need a fallback model or two deployments of the same model.
3. Set retries and cooldowns deliberately. Test them by forcing failures.
4. Log cost per call and compare it with the provider invoices for a week.
5. Move to the proxy only when central control matters.

Common Mistakes

• Assuming every provider feature carries over. Test the ones you rely on.
• Hardcoding model names across the codebase. Use aliases like "chat" and swap behind them.
• Using usage-based routing in production despite the docs' warning.
• Never testing the failure path. A fallback you haven't triggered is a hope, not a feature.
• Trusting the cost number blindly. It's an estimate until it matches an invoice.
• Swapping models without a regression check — a fallback deployment should still pass your eval set, not just return 200.
• And the one I learned the hard way. A launch day surge had the primary provider returning 429s within the hour; cooldown sidelined it in seconds and the fallback deployment carried the traffic — error rate stayed under 0.5%, and nobody but me noticed. What I hadn't tested: the fallback model had a smaller context window than the primary, and the longest support tickets came back politely truncated. The fallback worked, and it still failed, because I'd only ever tested that the wire was connected — not that the other side could do the job.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is LiteLLM and when should you use it?

LiteLLM is an open-source library that gives you a single, unified interface to call 100+ LLMs using the OpenAI request and response format. It comes as a Python SDK (import it, call models directly, with a Router for retries, fallbacks, and load balancing) and as a proxy — a self-hosted OpenAI-compatible gateway that adds virtual keys, budgets, centralized logging, guardrails, caching, and an admin UI for teams. The trade is one more dependency and possible lag behind provider-specific features. It's worth it once you use two or more providers; for a single provider, the official SDK is simpler.

How does the LiteLLM Router handle failures?

The Router takes a model_list where entries share a model_name alias (so identical aliases load-balance together). You rank deployments with priority order — lower values are higher priority — and when one fails the Router moves to the next level, with each level getting its own retries, then falls through to configured fallbacks. Rate-limit errors (429s) get exponential backoff; generic errors retry immediately. Failing deployments get sidelined: defaults are allowed_fails=3 and cooldown_time=5s, with immediate cooldown on a 429, a failure rate above 50% in the current minute, and non-retryable errors like 401, 404, 408. Note the docs are inconsistent on cooldown duration (one passage says "a minute" while the default says 5 seconds) — check the real value in your config and test it.

How do you track LLM cost with LiteLLM?

LiteLLM returns token usage by default and adds a response_cost on every call — in one example, response._hidden_params['response_cost']. Two helpers cover the rest: completion_cost(completion_response=response) returns total USD as a float, and cost_per_token(...) returns prompt and completion costs separately. The prices come from LiteLLM's own list, which is third-party data — treat the numbers as estimates and compare them with your provider's bill, ideally for a full week before trusting any dashboard built on them.

Should you use LiteLLM's proxy or SDK?

Use the SDK when one app needs to talk to several models: import it, swap the provider-prefixed model string, done. Use the proxy when many apps or teams share access and someone needs to track spend: it's a self-hosted, OpenAI-compatible gateway, so any client that works with OpenAI works with it with no code changes, and it adds virtual keys, budgets, centralized logging, guardrails, caching, and an admin UI. The cost is another service to run, secure, and monitor — move to it only when central control matters.

Wrapping This Up

LiteLLM gives you one call shape, one response shape and one error vocabulary across many providers, plus a Router for retries, fallbacks and load balancing. The proxy adds team-level control. The trade is another dependency and possible lag on provider-specific features — and remember that none of the code in this tutorial ran in this sandbox, so treat the docs' claims as a starting point for your own ten-minute test.

Install it in a test project, run the same prompt against two providers, then force a failure and watch the fallback. That ten-minute experiment tells you more than this article can — especially if you check whether the fallback can actually do the job, not just return a response.

Related Articles

• What Is LLMOps? Complete Guide to Running LLM Apps in Production
• LLMOps vs MLOps: What Changes When You Ship LLM Applications
• Best LLM APIs for Building Agents: Pricing and Performance
• Observability for AI Agents: Tracing with LangSmith and Langfuse