LLM Token Limits: The Settings That Actually Cut Spend

Contents

Figure 1: LLM Token Limits: The Settings That Actually Cut Spend — every request crosses the gateway, so every token can be counted there

Your model bill climbs, and nobody can say why. One team sends 8,000-token system prompts. Another retries failed calls in a tight loop. A third asks the same question 400 times a day.

An LLM gateway sits in the perfect spot to fix all of that. This guide explains how to set token usage limits and optimize token spend with an LLM gateway, using LiteLLM's current docs as the concrete example. The ideas carry over to any gateway that supports limits, caching and routing.

Why the Gateway Is the Right Place for Token Control

Every request passes through the gateway, so it sees tokens that app code never reports. It can count them, cap them, cache them and reroute them before the provider bills you. Doing this inside each app means fixing the same problem in ten codebases.

Control	What it does	Where it lives
Limits	Stops any key, team or user from consuming more than their share	tpm_limit, rpm_limit, max_parallel_requests, max_budget [1]
Reduction	Cuts the number of tokens you send and receive	max_tokens on requests, shorter prompts, caching
Routing	Sends each request to the cheapest model that can handle it	Model aliases, fallbacks, budget fallbacks [5]

Start with limits, because they protect you while you work on the rest.

Step 1: Measure Before You Optimize

You cannot trim what you cannot see. LiteLLM tracks spend per key, team and user when you run it with a database, and its responses include headers such as x-litellm-key-remaining-tokens-{model} and x-litellm-key-remaining-requests-{model} for per-model limits [1].

Pull a week of data and answer four questions:

1. Which keys and teams spend the most?
2. Which models eat the budget?
3. How many input tokens does a typical request carry, versus output tokens?
4. Which prompts repeat?

In many setups, a handful of keys account for most of the spend. Go after those first.

This works best when each app has its own virtual key, so attribution stays clean. If you have not set that up yet, the virtual keys and budgets guide covers the key-per-app pattern.

Step 2: Set Token Limits That Protect You

Limits keep one runaway job from hurting everyone else. LiteLLM supports tpm_limit, rpm_limit and max_parallel_requests on keys, users and teams [1].

curl --location 'http://localhost:4000/key/generate' \
  --header 'Authorization: Bearer <your-master-key>' \
  --header 'Content-Type: application/json' \
  --data '{"tpm_limit": 50000, "rpm_limit": 60, "max_parallel_requests": 10}'

Several details from the docs decide whether these numbers hold [1][2]:

Setting	What it changes
token_rate_limit_type	TPM counts total tokens by default; set input or output to count one side only [2]
model_tpm_limit, model_rpm_limit	Per-model ceilings on a key, team, org or project, so an expensive model gets a tighter cap [1]
disable_fallbacks_on_per_model_rate_limits	Returns 429 on a per-model limit instead of retrying a fallback model [2]
fail_closed_rate_limit_enforcement	Rejects with 503 when Redis counters are unreachable, instead of enforcing per instance from memory [2]
global_max_parallel_requests	Caps concurrency across the whole proxy, complementing the per-deployment max_parallel_requests [2]

The Redis point is the one that quietly breaks limits. Without shared counters, each replica counts separately, so a 50,000 TPM limit becomes 50,000 times your replica count. That is not a limit, it is a suggestion with extra steps.

Give the Gateway Better Guesses About Output Size

TPM limits work by reserving tokens before a call and reconciling after it. When a request omits max_tokens, the gateway has to guess how much output to reserve.

LiteLLM lets you declare that guess with default_estimated_output_tokens and a per-model map, default_estimated_output_tokens_per_model, on a key or a team [1]:

curl --location 'http://0.0.0.0:4000/key/generate' \
  --header "Authorization: Bearer $LITELLM_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "tpm_limit": 50000,
    "default_estimated_output_tokens": 600,
    "default_estimated_output_tokens_per_model": {"gpt-reasoning": 4000}
  }'

The docs advise sizing the estimate near the 95th percentile of what the model actually emits, not its maximum context [1]. Both directions have a cost:

• Declare too high and the over-reservation blocks requests the budget could have served.
• Declare too low and concurrent requests are admitted against an under-reservation, so the window overruns the limit once they finish.

The docs are explicit that the second failure is worse [1].

There is a second reservation model for the newer per-direction limits. The beta itpm/otpm deployment limits enforce input and output tokens per minute separately, and they resolve an "effective output cap" in a fixed order: the request's max_tokens or max_completion_tokens, then the model's max_output_tokens, then a hard default of 4096 [4]. A request with no max_tokens against a 32,768-output model reserves 32,768 tokens of headroom before it runs [4]. That single missing parameter is why concurrency collapses on some deployments.

Add Dollar Budgets on Top

Token limits control speed, and budgets control total spend. Pair them.

A key can carry a max_budget with a budget_duration, and the docs describe stacking several budget windows, such as a daily cap plus a monthly cap [1]:

curl 'http://0.0.0.0:4000/key/generate' \
  --header 'Authorization: Bearer <your-master-key>' \
  --header 'Content-Type: application/json' \
  --data '{
    "max_budget": 100,
    "budget_duration": "30d",
    "budget_limits": [
      {"budget_duration": "24h", "max_budget": 10},
      {"budget_duration": "30d", "max_budget": 100}
    ]
  }'

The reasoning in the docs is worth repeating: a single window cannot stop a bad day from burning your whole month [1].

Two operational catches:

1. Budgets need a database, or they silently do nothing. No spend rows, no enforcement.
2. Per-model key budgets (model_max_budget) are an Enterprise feature, so the budget fallback chain built on top of them needs that licence too [1][5].

Budget reservation also leans on max_tokens. LiteLLM estimates a request's maximum cost, reserves it and rejects the call up front if the reservation exceeds the budget [1]. A request with a sensible max_tokens reserves less, so it fails less often.

Step 3: Cap Output Tokens

Output tokens usually cost more than input tokens, and models love to ramble. Set max_tokens (or max_completion_tokens) on every request that has a reasonable ceiling. A summarizer that never needs more than 300 tokens should never receive permission to produce 4,000.

The gateway can help enforce this in three ways:

• Teach your teams to set the parameter, and lint for it in code review.
• Use the key-level estimate settings above so the reservation matches reality.
• Watch for prompts that ask for long, formatted answers when a short answer would do.

This is also the cheapest change on the list. No architecture, no new service, one parameter per call site.

Step 4: Reduce Input Tokens

Input waste hides in plain sight. Look for these patterns in your logs:

Pattern	Symptom in the logs	Fix
Bloated system prompts	A prompt that grew by accretion over six months	Rewrite it; it often halves without losing quality
Unbounded chat history	Input tokens growing every turn	Trim or summarise old turns
Repeated context	The same documents attached to every request	Reference them once, or cache the prefix
Verbose tool definitions	Dozens of schemas sent on calls that use none	Send only the tools that call needs

The gateway cannot rewrite your prompts for you, but it can show you the numbers. Sort requests by input size and fix the top offenders first.

LiteLLM's docs also describe prompt compression through a tool called Headroom, which I did not test, so evaluate it carefully before you rely on it [3].

Step 5: Cache What Repeats

The cheapest token is the one you never send. LiteLLM caches responses at the gateway, and the docs list several backends [3]:

Type	Backend	Use when
redis	Redis, Valkey, ElastiCache, Memorystore	Default for anything past a single worker
redis-semantic, valkey-semantic, qdrant-semantic	Vector similarity	Single-shot prompts with paraphrased repeats
s3, gcs	Object storage	Large, cold cache you can share
local, disk	In-process or on-disk	Local development only

litellm_settings:
  cache: true
  cache_params:
    type: redis

Set the connection through REDIS_URL, or REDIS_HOST, REDIS_PORT and REDIS_PASSWORD [3]. The proxy exposes /cache/ping to confirm the cache works [3].

The multi-worker trap: an in-memory cache lives inside one worker process, so four workers keep four separate caches and the hit rate drops roughly by the worker count [3]. Use Redis once you run more than one worker or replica.

Exact-Match Caching

Exact-match caching returns a stored answer when a request matches a previous one. It fits deterministic workloads: classification, extraction and FAQ-style questions. It misses when anything in the request changes, so small prompt tweaks reduce your hit rate.

Semantic Caching

Semantic caching embeds the prompt and serves the closest stored answer whose similarity clears a threshold [3].

litellm_settings:
  cache: true
  cache_params:
    type: redis-semantic
    similarity_threshold: 0.8
    redis_semantic_cache_embedding_model: my-embedding-model

This feature has sharp edges, and the docs say so plainly [3]:

• A semantic cache embeds the text of the entire messages array, system prompt included.
• On multi-turn or agentic traffic, each turn is the previous request plus a small append, so it can replay stale responses. Raising the threshold does not reliably fix this.
• Assistant tool_calls are not part of the embedded text, which makes consecutive agent turns even harder to tell apart.
• The recommended fix is exclusion: set "cache": {"no-cache": true} in the metadata of the keys your agents use, or make caching opt-in with mode: default_off.
• The embedding call runs inline before the request reaches the model, capped at 5 seconds. Past the deadline the lookup is abandoned, the response carries x-litellm-semantic-similarity: 0.0, and the request proceeds as a cache miss.

Start with exact-match caching on safe workloads. Add semantic caching only after you test a sample of its answers by hand.

Provider Prompt Caching

Providers such as OpenAI and Anthropic offer their own prompt caching, which discounts repeated prompt prefixes. LiteLLM's docs treat that as a separate feature from the gateway cache. Put stable content, like system instructions and reference documents, at the start of your prompts so the provider can reuse it. Check each provider's current rules and discounts, since they change.

Step 6: Route Easy Work to Cheaper Models

Not every request needs your best model. Classification, extraction and short rewrites often work fine on a small, cheap model. Define model aliases in the gateway, like fast-chat and smart-chat, and let each app pick the cheapest alias that passes its quality tests.

Then add safety nets [1][5]:

Mechanism	What it does	Caveat
Router fallbacks	Sends traffic to a backup model when the first fails or hits a limit	enforce_fallback_model_access defaults to true: the key must be allowed to call the fallback model [2]
Budget fallbacks	Reroutes to the next model in a per-key chain when model_max_budget is exhausted	Needs the Enterprise per-model budget, and only fires on that check [5]
enforce_fallback_budget	Defaults to true, so fallbacks stop once the key is out of budget	Set false to let fallbacks run anyway [2]

Budget fallbacks are more subtle than they look. They only apply to the per-key model_max_budget check inside the auth layer, spend is attributed to the fallback model, and router-level fallbacks keep running separately on provider errors [5]. If every fallback in the chain is over budget, the original BudgetExceededError surfaces [5].

This is where the load balancing guide pays off: routing decisions and spend decisions are the same decisions.

Step 7: Control Retries and Failure Behavior

Retries burn tokens quietly. A client that retries a long, failing request five times pays for five attempts if the provider bills partial usage.

1. Set sensible num_retries and request_timeout values at the gateway, not in every client.
2. Teach clients to back off after a 429 instead of hammering the endpoint.
3. Test what happens when a provider returns errors, since cost surprises often hide in failure paths.
4. Watch the retry path against your per-model limits, because fallbacks can turn one rejected request into a paid call on another model.

The timeout side of this deserves its own settings pass — the HTTP timeouts and connection tuning guide covers the values that stop a hung provider from holding a slot.

A Simple Optimization Loop

Run this loop every month:

1. Measure: rank keys, models and prompts by token spend.
2. Pick the biggest target: fix the top one, not the easiest one.
3. Change one thing: cap output, shorten a prompt, add a cache or switch a model.
4. Check quality: compare answers before and after, using a fixed test set.
5. Measure again: confirm the savings in real data.

Quality checks matter most. A cheaper answer that users hate costs more in the end.

Common Pitfalls

1. Setting budgets without a database. The limits never fire.
2. Skipping max_tokens. Output length drifts upward and nobody notices, and the reservation model inflates every concurrency check.
3. Turning on semantic caching for chat agents. Stale replays follow.
4. Running multiple gateway instances without Redis. Your effective limit multiplies by the instance count.
5. Optimizing without a quality test set. You save tokens and lose users.
6. Declaring output estimates far above real usage. You throttle traffic you could serve.
7. Forgetting that per-model key budgets are Enterprise. The budget fallback chain never fires on the free tier.
8. Assuming a fallback respects your per-model cap. By default it does not; set disable_fallbacks_on_per_model_rate_limits.

Wrapping Up

Gateways give you three levers for token control: limits to protect you, reduction to shrink what you send, and routing to pay less per request. Measure first, set limits second, then cap outputs, cache what repeats and route easy work to cheaper models. Test every change against real answers.

The failure modes repeat across teams: no database behind the budgets, no max_tokens on the calls, semantic caching on agents, and Redis missing across replicas. Fix those four and most bill surprises disappear.

So pull one week of usage by key today and find your single biggest spender. Fixing that one key will probably teach you more than any general advice, including this guide.

If you are still choosing a gateway, the LiteLLM vs OpenRouter vs Portkey comparison covers which of these controls you get on each platform.

Frequently Asked Questions

How do I set a token limit per API key in LiteLLM?

Send tpm_limit, rpm_limit and max_parallel_requests to /key/generate, or set the same fields on a team or user so the limit survives key rotation. Add model_tpm_limit and model_rpm_limit to give an expensive model a tighter ceiling than a cheap one. In a multi-replica deployment these counters must live in Redis, otherwise each instance counts separately and your real limit multiplies by the instance count.

Why do my LiteLLM budgets not block anything?

Budgets need a database. Without one, the proxy cannot track spend and the limit fails open. Once a database is connected, max_budget with a budget_duration resets on a schedule, and budget_limits lets you stack several windows, such as a daily cap plus a monthly cap on the same key. Check that the key actually has a budget and that the spend row is being written before you trust the cap.

What is default_estimated_output_tokens in LiteLLM?

It is the output size the gateway assumes when a request does not declare max_tokens, used to reserve token-per-minute headroom before the call and reconcile after it. You can set one value, or a per-model map with default_estimated_output_tokens_per_model, on a key or a team. Size it near the 95th percentile of what the model actually emits: too high throttles traffic you could have served, and too low admits concurrent requests against an under-reservation that overruns the limit.

Is semantic caching safe for chat agents in LiteLLM?

The docs say no, and they say it plainly. A semantic cache embeds the text of the entire messages array, system prompt included, and serves the closest cached response above the similarity threshold. On multi-turn or agentic traffic each turn is the previous request plus a small append, so it replays stale responses. Keep it for single-shot prompts, exclude agent traffic with cache.no-cache metadata on those keys, or use exact-match caching instead.

Does LiteLLM caching work across multiple workers?

Not with the in-memory cache: it lives inside one worker process, so four workers keep four separate caches and the hit rate drops roughly by the worker count. Use Redis or Valkey once you run more than one worker or replica. The docs list redis, valkey, redis-semantic, valkey-semantic, qdrant-semantic, s3, gcs, local and disk as cache types, and the proxy exposes /cache/ping to prove the backend is reachable.

How do I stop fallbacks from bypassing per-model rate limits?

Set disable_fallbacks_on_per_model_rate_limits: true in general_settings. By default a request that exceeds a key, team, organisation or project per-model limit can retry on a router fallback, which spends money on a different model instead of returning 429. The flag makes that per-model check a hard cap. Other local limits, such as a key's overall rpm_limit, keep their existing fallback behaviour.

Sources

• [1] LiteLLM, Budgets and rate limits for keys, teams and users — tpm_limit, rpm_limit and max_parallel_requests on keys, users and teams, model_rpm_limit and model_tpm_limit per-model ceilings, the x-litellm-key-remaining-tokens-{model} and x-litellm-key-remaining-requests-{model} response headers, default_estimated_output_tokens and default_estimated_output_tokens_per_model on keys and teams with the 95th-percentile guidance and both over- and under-reservation failure modes, max_budget with budget_duration, the stacked budget_limits windows with the "$10/day and $100/month" example, budget reservation using the request's maximum cost, and model_max_budget as an Enterprise feature: docs.litellm.ai/docs/proxy/users
• [2] LiteLLM, Config settings — token_rate_limit_type with its total default and input/output options, disable_fallbacks_on_per_model_rate_limits and the note that other local limits keep their fallback behaviour, fail_closed_rate_limit_enforcement returning 503 when Redis counters are unreachable, max_parallel_requests and global_max_parallel_requests, enforce_fallback_model_access defaulting to true, and enforce_fallback_budget defaulting to true: docs.litellm.ai/docs/proxy/config_settings
• [3] LiteLLM, Caching — the cache type table (redis, valkey, redis-semantic, valkey-semantic, qdrant-semantic, s3, gcs, local, disk), the warning that an in-memory cache is per worker so hit rate drops by the worker count, the /cache/ping endpoint, REDIS_URL and the REDIS_HOST/REDIS_PORT/REDIS_PASSWORD variables, semantic caching's embedding of the whole messages array, the multi-turn and agentic staleness warning, cache.no-cache key metadata and mode: default_off exclusion, the 5-second inline embedding cap with the x-litellm-semantic-similarity: 0.0 fallback, and the Headroom prompt compression page: docs.litellm.ai/docs/proxy/caching, docs.litellm.ai/docs/proxy/caching_semantic and docs.litellm.ai/docs/proxy/headroom
• [4] LiteLLM, Separate ITPM / OTPM rate limits (beta) — per-deployment itpm and otpm with the enforce_model_rate_limits pre-call check, the reservation-then-reconcile model, the effective output cap order (request max_tokens/max_completion_tokens, then model max_output_tokens, then a 4096 default), the worked example reserving 32,780 tokens for a 150-token answer, ITPM refunds on failure with no OTPM charge, cached prompt tokens excluded after reconciliation, and the x-ratelimit-*-input-tokens / x-ratelimit-*-output-tokens headers: docs.litellm.ai/docs/proxy/io_token_rate_limits
• [5] LiteLLM, Budget fallbacks (v1.92.0+) — rerouting on a key's model_max_budget instead of returning budget_exceeded, the budget_fallbacks map keyed by primary model, spend attributed to the fallback model, router-level fallbacks staying independent, and BudgetExceededError when every fallback is over budget: docs.litellm.ai/docs/proxy/budget_fallbacks

Related Articles

• LiteLLM Virtual Keys and Budgets: The Bill Nobody Watches
• LiteLLM vs OpenRouter vs Portkey: Pick the Wrong One
• LiteLLM Load Balancing: Your RPM Limit Is a Suggestion
• LiteLLM Timeouts: The Setting Your Load Balancer Kills