LiteLLM Proxy Server: Complete Guide to Running One API for Every LLM Provider

Contents

Figure 1: LiteLLM Proxy Server: Complete Guide to Running One API for Every LLM Provider — one endpoint in front of many providers, and one place you have to guard

Your app calls OpenAI today, Anthropic tomorrow, and a self-hosted model next week. Each switch means new SDKs, new error formats, and new keys scattered across the codebase. What if every model answered through one endpoint?

That is the pitch of the LiteLLM Proxy Server, an open-source AI gateway that speaks the OpenAI API format and routes requests to 100+ providers [1]. This guide walks you through setup, keys, budgets, load balancing, and fallbacks. It also covers a serious security incident you need to know about before you install anything.

What the LiteLLM Proxy Does

LiteLLM Proxy sits between your applications and your model providers. Your code sends OpenAI-style requests to the proxy, and the proxy translates them for whichever provider you configured. The official docs describe it as OpenAI-compatible, so any OpenAI SDK works when you point it at the proxy's URL [1].

Here is what you get from one gateway:

Capability	What it buys you
Unified interface	One request format for many providers
Virtual keys	Each app or teammate gets its own key, not a raw provider key
Spend tracking and budgets	Cost per key, visible and capped
Load balancing	Traffic spread across several deployments of one model
Fallbacks	Automatic failover to a backup model when a provider breaks
Admin UI	Models, keys and a test playground in the browser

Why bother? Because your provider keys stay in one place, your costs show up in one dashboard, and swapping a model becomes a config change instead of a code change.

Read This First: The March 2026 Supply Chain Attack

On 24 March 2026, attackers published two malicious LiteLLM versions to PyPI: 1.82.7 and 1.82.8. Multiple security firms reported that the packages stole credentials and sensitive system data and sent them to an attacker-controlled server [5]. Version 1.82.7 hid its payload in proxy_server.py, and 1.82.8 added a litellm_init.pth file for persistence [5].

PyPI quarantined the package and removed both versions, and reports say 1.82.6 and earlier stayed clean [5]. LiteLLM's own advisory said it believed the compromise started with a Trivy dependency in its CI/CD security scanning [5]. Security write-ups also pointed out that a LiteLLM proxy holds your provider keys, so a compromise there exposes your whole LLM surface.

Two details from the advisory change how you should read that story:

1. The malicious window was short. The packages were live from 10:39 UTC and quarantined within about forty minutes, and no malicious code reached the main branch [5].
2. The official proxy Docker image was not affected. That deployment path pins dependencies in requirements.txt and does not consume the compromised PyPI packages, so Docker users were explicitly listed as unaffected [5].

What should you do?

1. Check your history. Search CI logs and lockfiles for litellm-1.82.7 or litellm-1.82.8.
2. Search for the persistence file. Run find "$HOME" -name "litellm_init.pth" 2>/dev/null — in 1.82.8 that file executes on every Python start, without anyone importing LiteLLM.
3. Check for the callbacks. Outbound traffic to models.litellm.cloud or checkmarx.zone is an indicator of compromise; neither domain belongs to LiteLLM [5].
4. Rotate secrets if you installed either version, even briefly. Treat every key on that machine as exposed.
5. Pin exact versions and use lockfiles. Never install an unpinned latest in production.
6. Read the official advisory from LiteLLM and recheck current guidance before you deploy [5].

This incident does not mean the project is unusable today. LiteLLM shipped 1.83.0 through a rebuilt release pipeline with isolated environments and stronger gates [5]. It does mean you should treat your gateway like the high-value target it is.

Option 1: Docker Compose With a Database (Recommended)

The official quickstart runs the gateway and a Postgres database together [2]. The database stores your models, keys, and spend logs. You need it for virtual keys and budgets.

curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker/docker-compose.quickstart.yml
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
docker compose -f docker-compose.quickstart.yml up -d

The gateway starts on port 4000. Open the compose file before you run it, and pin a specific release tag instead of a floating one. The docs themselves note that you can edit the file to pin a release [2].

Understand the Two Keys

The setup creates two secrets, and mixing them up causes pain:

Variable	Role	What happens if you get it wrong
LITELLM_MASTER_KEY	Root credential for every management call, and the Admin UI password by default	Anyone holding it has full admin access
LITELLM_SALT_KEY	Encrypts the provider keys you store	No in-place rotation, and changing it makes stored credentials unreadable

Treat the master key like a root password, and back up your .env file [2].

Log In and Add a Model

1. Open http://localhost:4000/ui in your browser.
2. Log in as admin with your master key as the password.
3. Go to Models + Endpoints, open Add Model, pick your provider, paste your API key.
4. Click Test Connect before you save.

The docs also offer a one-line install script that you pipe into a shell [2]. I would skip that shortcut for anything beyond a throwaway laptop test. After March's incident, read a script before you run it.

Option 2: A Config File Without a Database

If you only need the OpenAI-compatible API, you can skip the database. Write a config file and run the plain image.

# litellm_config.yaml
model_list:
  - model_name: fast-chat
    litellm_params:
      model: openai/<your-openai-model>
      api_key: os.environ/OPENAI_API_KEY
  - model_name: smart-chat
    litellm_params:
      model: anthropic/<your-anthropic-model>
      api_key: os.environ/ANTHROPIC_API_KEY

docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=<your-openai-key> \
  -e ANTHROPIC_API_KEY=<your-anthropic-key> \
  -e LITELLM_MASTER_KEY=sk-<paste-a-long-random-key> \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:<pinned-tag> \
  --config /app/config.yaml

The model_name is the alias your apps use, and litellm_params.model tells LiteLLM which provider model to call [3]. The os.environ/ prefix pulls keys from environment variables, so they stay out of the file. Replace the placeholders with real model names from your providers' docs.

There is a catch you must know about. The docs warn that budgets do not work without a database [2]. A configured max_budget never fires on this path, and the proxy keeps serving requests past the limit with no error. Virtual keys also fail without a database. If cost control matters, run Postgres, or set spending limits at the provider.

Call the Proxy From Your App

The proxy accepts standard OpenAI requests. Point the SDK at your gateway and use a virtual key.

from openai import OpenAI

client = OpenAI(
    api_key='sk-<your-virtual-key>',
    base_url='http://localhost:4000',
)

response = client.chat.completions.create(
    model='smart-chat',   # your alias from the config
    messages=[{'role': 'user', 'content': 'Say hello in five words.'}],
)
print(response.choices[0].message.content)

To switch providers, change the model string to another alias. Nothing else in your code moves. I could not run this snippet in my environment, so test it against your own gateway.

Virtual Keys, Budgets, and Rate Limits

Never hand your master key or provider keys to an application. Create virtual keys instead. Each virtual key can carry its own budget, rate limits, and model access, and the proxy tracks its spend automatically [4].

The easiest route is the Admin UI: open Virtual Keys, click + Create New Key, and copy the key right away, because the UI shows it only once. You can also call the /key/generate endpoint with the master key [4]:

curl -X POST 'http://localhost:4000/key/generate' \
  -H 'Authorization: Bearer <your-master-key>' \
  -H 'Content-Type: application/json' \
  -d '{"models": ["fast-chat"], "max_budget": 10, "rpm_limit": 60}'

Check the Virtual Keys docs for the current list of parameters, since fields and defaults change between releases [4]. Good habits include one key per application, short lifetimes for experiments, and tight model lists for each key.

Key hygiene rule	Why it pays off
One key per application	Revoking one app never breaks another
Short lifetimes for experiments	Forgotten keys stop being a liability
Tight models list per key	A cheap model cannot quietly become an expensive one
Budget per key	Runaway loops hit a wall instead of an invoice

Load Balancing Across Deployments

Want more throughput, or a second region as insurance? Add several deployments under the same model_name. The proxy's router spreads requests across them, and the docs say it uses a simple-shuffle strategy by default [3]. They also recommend setting rpm per deployment to maximise throughput [3].

model_list:
  - model_name: smart-chat
    litellm_params:
      model: azure/<deployment-one>
      api_base: os.environ/AZURE_API_BASE_ONE
      api_key: os.environ/AZURE_API_KEY_ONE
      rpm: 60
  - model_name: smart-chat
    litellm_params:
      model: azure/<deployment-two>
      api_base: os.environ/AZURE_API_BASE_TWO
      api_key: os.environ/AZURE_API_KEY_TWO
      rpm: 60

Your apps still ask for smart-chat. They never learn that two deployments answer.

Fallbacks, Retries, and Cooldowns

Providers fail. Rate limits hit, regions go down, and models time out. Fallbacks tell the proxy what to try next, after it exhausts its retries [3].

litellm_settings:
  num_retries: 3
  request_timeout: 10
  fallbacks: [{"smart-chat": ["fast-chat"]}]
  allowed_fails: 3
  cooldown_time: 30

Here is what each setting does, according to the docs [3]:

Setting	Behaviour
num_retries	Retries each model group before falling back
request_timeout	Raises a timeout error when a call takes too long
fallbacks	Maps one model alias to an ordered list of backups
allowed_fails + cooldown_time	Parks a failing model for a while after repeated failures

The docs describe three fallback types: general fallbacks for most errors such as rate limits, context_window_fallbacks for prompts that exceed a model's limit, and content_policy_fallbacks for policy rejections [3].

One more detail matters when you fall back across providers: encrypted reasoning from the failed deployment cannot move to a different provider, so the backup model reasons fresh on that turn.

More Features Worth Knowing

The docs list several other gateway features [1]:

• Caching, to skip a provider call for repeated prompts.
• Guardrails, for input and output checks before traffic reaches a model.
• Logging and alerting, to see spend and failures as they happen.
• An MCP gateway, for governed tool access from agents.

I did not test them for this guide, so read each feature's page before you rely on it. Start with logging and spend tracking, because visibility pays off fastest.

Production Checklist

Before real traffic arrives, run through this list:

1. Pin the version. Choose a reviewed release tag, and update on purpose.
2. Use a database. You need it for keys, budgets, and spend tracking.
3. Guard the master key. Keep it out of source control, and rotate it if it leaks.
4. Back up the salt key. Losing or changing it locks you out of stored provider keys.
5. Put TLS in front. Do not expose plain HTTP to the internet.
6. Restrict network access. Your gateway should not accept traffic from everywhere.
7. Set provider-side limits too. A second spending cap catches mistakes the proxy misses.
8. Follow the official production guide. The docs cover Helm, Terraform, and Kubernetes deployments [2].

Common Pitfalls

I see these mistakes constantly:

1. Installing an unpinned latest. The March incident shows why that is risky.
2. Setting a budget with no database. The budget silently does nothing.
3. Sharing the master key with apps. Create virtual keys instead.
4. Changing the salt key. Stored credentials become unreadable.
5. Skipping fallbacks. One provider outage then takes your whole app down.
6. Ignoring spend logs. You find out about runaway costs on the invoice.
7. Exposing port 4000 directly to the internet. The gateway holds every provider key you own, which makes it a more attractive target than the app behind it.
8. Treating the proxy as observability. A gateway tells you spend and errors, not whether your answers improved.

Wrapping Up

The LiteLLM Proxy gives you one OpenAI-compatible endpoint for many providers, plus virtual keys, budgets, load balancing, and fallbacks. Run it with a database, pin your versions, guard your master key, and treat the March 2026 incident as a standing reminder to verify what you install.

So spin up the Docker Compose quickstart on a test machine, add one model, and send a single request through the gateway. Then add a second provider and a fallback, and watch your code stay exactly the same.

If you are still deciding where the gateway sits in your stack, LLMOps vs MLOps: what changes covers the operational split, and multi-cloud MLOps without vendor lock-in is the same argument applied to clouds rather than providers.

Frequently Asked Questions

What does the LiteLLM Proxy Server actually do?

It sits between your applications and your model providers. Your code sends OpenAI-format requests to the proxy, and the proxy translates them for whichever provider you configured, so any OpenAI SDK works once you point it at the gateway URL. On top of routing you get virtual keys per app or teammate, spend tracking and budgets, load balancing across deployments, fallbacks when a provider breaks, and an admin UI for models, keys and a test playground.

Was the LiteLLM PyPI compromise in March 2026 real, and am I affected?

It is real. Versions 1.82.7 and 1.82.8 were published to PyPI on 24 March 2026 with credential-harvesting malware and were quarantined within about forty minutes. You are affected if you installed or upgraded LiteLLM via pip in that window without pinning, or if an unpinned transitive dependency pulled it in. Users running the official proxy Docker image were not impacted, because that path pins dependencies in requirements.txt. If either version reached a machine, rotate every secret it could read.

Do I need Postgres to run the LiteLLM proxy?

Not for plain OpenAI-compatible routing, but you do need it for virtual keys, budgets and spend tracking. The docs are explicit that budgets do not work without a database: a configured max_budget never fires, and the proxy keeps serving requests past the limit with no error. If cost control matters, run the Docker Compose quickstart with Postgres, or enforce spending limits on the provider side as well.

What is the difference between LITELLM_MASTER_KEY and LITELLM_SALT_KEY?

The master key is the root credential: it authorises every management call and doubles as the admin UI password by default, so anyone holding it has full admin access. The salt key encrypts the provider keys you store, has no in-place rotation, and changing it makes stored credentials unreadable. Treat the master key like a root password, never ship it to an application, and back up the .env file holding the salt key.

How do fallbacks and cooldowns work in LiteLLM?

Fallbacks map one model alias to an ordered list of backups, tried after num_retries is exhausted per model group. allowed_fails and cooldown_time park a failing deployment for a fixed period so traffic stops hammering it. The docs describe three fallback types: general fallbacks for errors like rate limits, context_window_fallbacks for prompts that exceed a model's context, and content_policy_fallbacks for policy rejections.

Can I load balance across two Azure or Bedrock deployments with the same model name?

Yes. List several deployments under the same model_name, and the router spreads requests across them, using a simple-shuffle strategy by default. Set rpm per deployment as the docs recommend, so the router knows each deployment's capacity and maximises throughput. Your applications keep asking for the shared alias and never learn that more than one deployment is answering.

Sources

• [1] LiteLLM, official documentation — the OpenAI-compatible gateway, the provider list, and the feature set covering caching, guardrails, logging and the MCP gateway: docs.litellm.ai
• [2] LiteLLM, proxy quickstart and production guide — the Docker Compose quickstart with Postgres, pinning a release tag in the compose file, the one-line install script, the database requirement for budgets, and the Helm/Terraform/Kubernetes deployment paths: docs.litellm.ai/docs/proxy/quick_start · deploy
• [3] LiteLLM, config and routing docs — model_list, model_name aliases, litellm_params.model, the os.environ/ prefix for keeping keys out of config files, the simple-shuffle router with per-deployment rpm, and the num_retries, request_timeout, fallbacks, allowed_fails and cooldown_time settings with the three fallback types: configs · load balancing · reliability
• [4] LiteLLM, virtual keys — /key/generate, per-key budgets, rate limits and model access, plus the parameter list that changes between releases: docs.litellm.ai/docs/proxy/virtual_keys
• [5] LiteLLM, "Security Update: Suspected Supply Chain Incident" — versions 1.82.7 and 1.82.8, the 10:39 UTC publish window, the Trivy origin, the proxy_server.py and litellm_init.pth payloads, the models.litellm.cloud and checkmarx.zone indicators, the Docker image being unaffected, and the 1.83.0 re-release: docs.litellm.ai/blog/security-update-march-2026 · GHSA-5mg7-485q-xm76

Related Articles

• LLMOps vs MLOps: What Changes
• Multi-Cloud MLOps: Avoiding Vendor Lock-In
• Evaluating AI Agents: Metrics, Benchmarks, Test Strategies