LiteLLM Load Balancing: Your RPM Limit Is a Suggestion

Contents

Figure 1: LiteLLM Load Balancing: Your RPM Limit Is a Suggestion — three deployments, one model name, and a limit nobody enforces

One API key has a rate limit. Three deployments behind one name have three. That is the whole pitch for LiteLLM load balancing, and the setup really is that small. The hard part is picking a strategy and knowing what its settings actually do.

I did not run LiteLLM, since it is not installable here. Everything below comes from LiteLLM's docs. My script is a stand-in that models the documented rules and shows their effects.

The Basic Setup

In the proxy, add several model_list entries with the same model_name. The docs' example spreads requests for one model name across three Azure deployments, each with its own api_base, api_key and rpm [2]. Clients ask for the shared name and the router picks a deployment.

model_list:
  - model_name: gpt-6-luna
    litellm_params:
      model: azure/gpt-6-luna
      api_base: os.environ/AZURE_API_BASE_ONE
      api_key: os.environ/AZURE_API_KEY_ONE
      rpm: 60
  - model_name: gpt-6-luna
    litellm_params:
      model: azure/gpt-6-luna
      api_base: os.environ/AZURE_API_BASE_TWO
      api_key: os.environ/AZURE_API_KEY_TWO
      rpm: 60

router_settings:
  routing_strategy: simple-shuffle

Each entry's litellm_params can carry model, api_key, api_base, rpm, tpm, weight, order and max_parallel_requests. Optional model_info takes id, allowed_fails, cooldown_time and related settings [1]. Set the strategy under router_settings as routing_strategy.

To check it works, look at the model_id in the response headers. The docs say to use it to confirm requests are being balanced [2].

Pick a Strategy

The default is simple-shuffle, labelled "Weighted Pick" and marked RECOMMENDED on the routing page [1]. The docs recommend it for best performance in production [1].

Strategy	What the docs say	Watch out for
simple-shuffle (default)	Picks by rpm or tpm, or randomly if neither is set; weights supported	The recommended production choice
usage-based-routing-v2	Filters out deployments over their limit, picks lowest TPM use this minute; uses Redis	"Not recommended for production due to performance impacts"
latency-based-routing	Favours faster deployments, with ttl and lowest_latency_buffer settings	Default ttl is not stated
cost-based-routing	Routes to the cheapest deployment using input_cost_per_token and output_cost_per_token	Redis needs are not stated
least-busy	Routes to the deployment with the fewest active requests	Docs pitch it for high-concurrency work, with no formula

The proxy page's table marks usage-based routing as "bad for perf" [2]. That is the line to remember: the smartest-sounding strategy is the one the docs warn about.

Weights and rpm: Hints, Not Walls

With simple-shuffle, a weight of 9 is picked "90% of the time" per the docs [1]. My stand-in matched that: over 10,000 requests, the heavier deployment got 90.0%.

The subtler point is rpm and tpm. By default they only inform routing, and nothing stops a deployment from receiving more than its limit [2]. Setting enforce_model_rate_limits turns them into hard limits, and requests over the limit get a 429 before reaching the provider [2]. RPM is enforced exactly, but TPM is best-effort because token counts are only known after the response. Separate itpm and otpm settings exist for providers with distinct input and output limits [2].

In my simulation, 1,500 requests in one minute against a total rpm of 1,000 pushed every deployment past its quota in soft mode, with zero rejections. The hard limit capped each deployment at its rpm and rejected 500 requests with 429s. That is the trade: hard limits protect your quota but turn overload into errors, so pair them with fallbacks.

Priority Tiers With order

order sets priority: "Lower values = higher priority" [1]. Deployments with the same order are chosen among by the routing strategy [1]. Failures escalate to the next order level, and then to any configured fallbacks. It is the right tool for "use my reserved capacity first, then pay-as-you-go."

One behaviour makes priority practical: for 429 errors specifically, the failed deployment is immediately placed on cooldown, and if every order=1 deployment is cooling down, the router picks order=2 deployments during retries without waiting for the fallback path [2].

Latency-Based Routing and the Buffer

Latency routing favours the fastest deployment. The lowest_latency_buffer setting (the docs' example is 0.5) admits deployments within a 50% buffer of the fastest, so one deployment does not get all the traffic [1]. The exact formula is not given, so my simulation assumes "within 1.5 times the fastest." Under that assumption, a buffer of 0 sent all 3,000 requests to the fastest deployment, while 0.5 split them between the two near the front and still skipped the slow one.

The latency window itself is set with ttl under routing_strategy_args [1]. The page shows ttl: 10 in an example and never states a default, so set it rather than inheriting one.

Gotchas Worth Knowing

1. Limits are per process. The docs describe max_parallel_requests as a per-deployment concurrency cap and give the precedence chain: max_parallel_requests, then rpm, then int(tpm / 1000 * 6) with a minimum of 1, then the router's default_max_parallel_requests [1]. They do not say whether that counter is shared across workers or pods, so treat four workers with a limit of 10 as potentially 40 concurrent calls until you test it. Note the chain itself: setting rpm silently becomes a concurrency cap.
2. Multiple proxies need Redis. Set redis_host, redis_password and redis_port so rpm and tpm usage is tracked across instances [2]. Without it, usage is tracked per instance [3].
3. Failover has defaults you might not expect. The proxy page says "default retries are 3" but its own example config sets num_retries: 2 [2]. Set it explicitly.
4. A 429 cools the deployment down, and the proxy page gives no duration. The routing page documents a 5s default [1][2].
5. Pre-call checks filter deployments. enable_pre_call_checks skips deployments whose context window is too small, and with region_name ones outside a requested region [1].
6. Responses API caveat. With different API keys across deployments, encrypted reasoning items can fail to decrypt. The docs point to the encrypted_content_affinity check [1].

A Starter Recipe

1. Put every deployment of the same model under one model_name.
2. Keep simple-shuffle unless you have a measured reason to change.
3. Set accurate rpm and tpm, and turn on enforce_model_rate_limits if overload would hurt.
4. Use order for reserved-then-overflow capacity.
5. Configure Redis before you run more than one proxy.
6. Test failover with the docs' mock_testing_rate_limit_error: true and watch which model_id answers [4].

Common Pitfalls

1. Assuming rpm is enforced. It is a routing hint until enforce_model_rate_limits is on.
2. Setting rpm and expecting no concurrency cap. The precedence chain turns rpm into a parallel-request limit when nothing else is set.
3. Reaching for usage-based routing because it sounds smarter. The docs advise against it in production.
4. Leaving ttl unset on latency routing. No default is documented.
5. Running several proxies without Redis. Every instance counts on its own.
6. Trusting one retry number. The same page states 3 and configures 2.
7. Ignoring model_id when debugging balance. It is the only cheap evidence that traffic actually spread.
8. Enforcing limits without fallbacks. You converted a slow response into a hard 429.

Wrapping Up

LiteLLM load balancing is mostly configuration: shared model names, honest rpm numbers and one strategy that does what you need. Stay with the default unless you can measure a reason to leave it, and remember that limits that do not enforce are just suggestions.

Check what your config does under overload before your users do.

If you are still setting up the gateway, the LiteLLM proxy guide covers the quickstart, the retries guide covers what happens when a deployment fails, and the budgets guide covers the other half of limits.

Frequently Asked Questions

How does LiteLLM load balance across multiple deployments?

You add several model_list entries with the same model_name, each with its own api_base, api_key, rpm and optional weight. Clients ask for the shared name and the router picks a deployment according to routing_strategy. Check the model_id in the response headers to confirm requests are actually spread across deployments rather than landing on one.

Which LiteLLM routing strategy should I use in production?

simple-shuffle, the default. The docs recommend it for best performance in production because it picks by rpm or tpm or at random without extra lookups. usage-based-routing-v2 is explicitly not recommended for production due to performance impacts, since it adds significant latency from Redis operations. Use latency-based-routing or cost-based-routing only when you have measured a reason.

Are rpm and tpm hard limits in LiteLLM?

Not by default. rpm and tpm values are only used for routing decisions, picking deployments with capacity, so nothing stops a deployment from receiving more than its stated limit. Setting enforce_model_rate_limits turns them into hard limits, and requests over the limit are blocked with a 429 before reaching the provider. RPM is enforced exactly; TPM is best-effort because token counts are only known after the response.

What does order do in a LiteLLM model group?

order sets deployment priority, and lower values mean higher priority. Deployments sharing an order are chosen among by the routing strategy, and failures escalate to the next order level before reaching configured fallbacks. For 429 errors specifically, the failed deployment is immediately placed on cooldown, and if every order=1 deployment is cooling down the router picks order=2 during retries without waiting for the fallback path.

What is lowest_latency_buffer in latency-based routing?

It sets how wide a latency band counts as competitive. The docs example uses 0.5, meaning deployments within 50 percent of the fastest are candidates, so one fast deployment does not absorb every request. The exact formula is not published, and the default ttl for the latency averaging window is not stated either, so set both explicitly and watch the model_id distribution.

Why do my deployments exceed their rpm when I run several proxies?

Because the counter is local unless you share it. Set redis_host, redis_password and redis_port so rpm and tpm usage is tracked across instances; without Redis each instance tracks usage on its own, so a limit can effectively multiply by the number of proxies. The router docs also note that Redis backs the cooldown and usage state in production.

Sources

• [1] LiteLLM, Routing strategies — the recommendation of simple-shuffle for production, the Weighted Pick label, the usage-based warning, latency ttl and lowest_latency_buffer examples, order with lower values as higher priority, the max_parallel_requests precedence chain, enable_pre_call_checks context and region filtering, encrypted_content_affinity, the 5s cooldown default, and model_id from _hidden_params: docs.litellm.ai/docs/routing
• [2] LiteLLM, Proxy load balancing — the three-deployment config, the model_id header check, the strategy table marking usage-based routing as bad for perf, enforce_model_rate_limits turning rpm and tpm into hard limits with a 429 before the provider, itpm and otpm, redis_host/redis_password/redis_port, the "default retries are 3" line beside a config that sets num_retries: 2, and the 429-cooldown behaviour with order: docs.litellm.ai/docs/proxy/load_balancing
• [3] LiteLLM, Redis requirements — the shared state that keeps counters consistent across instances and what happens without it: docs.litellm.ai/docs/proxy/redis_requirements
• [4] LiteLLM, Fallbacks — mock_testing_rate_limit_error for exercising failover, and the note that mock_testing_* flags are stripped from proxy requests from v1.85.0 while still working for direct Router calls: docs.litellm.ai/docs/proxy/reliability

Related Articles

• LiteLLM Retries and Fallbacks: Stop Retrying a 401
• LiteLLM Errors: Your 401 Might Be a Budget Problem
• LiteLLM Virtual Keys and Budgets: Control Who Spends What