LiteLLM Retries and Fallbacks: Stop Retrying a 401

Contents

Figure 1: LiteLLM Retries and Fallbacks: Stop Retrying a 401 — a bad key and a rate limit are not the same problem

Your LLM call fails at 2 a.m. Should the code retry, switch models or give up? LiteLLM can do all three, but only if you tell it which error gets which treatment. A bad API key and a rate limit are not the same problem. Treating them the same wastes calls and time.

One honest note up front: LiteLLM is not installed here and PyPI is unreachable, so I never ran it. I read its docs and built a stand-in simulation of the documented rules. The settings below come from the docs. The timings in the simulation are my own assumptions.

What LiteLLM Gives You

LiteLLM maps provider errors to OpenAI-style exception classes and lets you import them from litellm, for example from litellm import BadRequestError [1]. The Router and proxy add retries, cooldowns and fallbacks on top.

The error classes you will meet most [1]:

Class	Meaning
401 AuthenticationError	The key is wrong or expired
402 PaymentRequiredError	The provider says you are out of credits; the Router cools that deployment down if its model group has another deployment
403 PermissionDeniedError	The key lacks access to this model
404 NotFoundError	Wrong model name or route
408 Timeout	The provider did not answer in time
422 UnprocessableEntityError	The request body is invalid
429 RateLimitError	You are sending too much
503 ServiceUnavailableError	The provider is down
400 ContextWindowExceededError	Prompt exceeds the model's context; has its own fallback list
400 ContentPolicyViolationError	Provider policy rejection; has its own fallback list

The docs' exception page says nothing about which codes are retryable. It calls litellm._should_retry(e.status_code) in an example but never says what that returns [1]. Read that gap carefully before you build on assumptions.

Retries: Set Them per Error Type

num_retries controls retries. The Router page shows Router(..., num_retries=3) but gives no default [2]. Precedence, highest first [2]:

Rank	Source
1	x-litellm-num-retries header (proxy only)
2	The request body
3	The deployment's litellm_params
4	litellm_settings

Setting 0 disables retries for a request.

The better tool is retry_policy. It sets retries per error type: AuthenticationErrorRetries, RateLimitErrorRetries, NotFoundErrorRetries and more, plus DefaultRetries for everything else [2]. The most specific field wins. The docs' own example sets AuthenticationErrorRetries to 0 [2].

router_settings:
  retry_policy:
    AuthenticationErrorRetries: 0
    RateLimitErrorRetries: 3
    ServiceUnavailableErrorRetries: 2
    NotFoundErrorRetries: 0
    DefaultRetries: 1

One trap in that mechanism: a field left unset defers to the next one, so a policy that only sets DefaultRetries retries 404s too. Set NotFoundErrorRetries: 0 if you want them left alone [2].

Why Zero Retries for a Bad Key

A wrong key does not fix itself in two seconds. In my simulation, one blanket num_retries=3 sent a bad key to the primary four times before falling back. With an auth retry limit of 0, it was sent once. Both ended at the backup, but the blanket policy added 7 seconds of simulated backoff for nothing. The backoff timings are my assumption, not LiteLLM's.

The real cost is not the seven seconds. It is that a 401 storm looks like provider instability in your logs, while the actual fault is a key nobody rotated.

Cooldowns: Stop Hitting a Dead Deployment

allowed_fails and cooldown_time handle the case where one deployment keeps failing [2].

Setting	Meaning	Documented default
allowed_fails	Failures a deployment gets before cooldown	3
cooldown_time	How long it sits out	5s (DEFAULT_COOLDOWN_TIME_SECONDS)

Both can be set per deployment, but only under model_info, because litellm_params is sent to the provider [2]. allowed_fails_policy sets failure counts per error type, such as RateLimitErrorAllowedFails [2].

The docs contradict themselves in two places. The Router page says a deployment is cooled down "for a minute" while the default is 5s [2]. The reliability page shows allowed_fails: 3 with a comment saying "fails > 1 call in a minute" [3]. Test the actual behaviour before you trust either number.

In my simulation, after 3 failures the primary was skipped, and all five requests still got answers from the backup. The primary was tried again once the cooldown expired. Whether LiteLLM skips a cooled-down deployment exactly this way is my assumption.

Fallbacks: Different Lists for Different Failures

The reliability page says that when a call "fails after num_retries", LiteLLM falls back to another model group, in the listed order [3]. It describes three kinds [3]:

List	Applies to
fallbacks	The remaining errors, such as RateLimitError, 429 and 500 responses
context_window_fallbacks	ContextWindowExceededError — pre-call checks must be enabled to exercise this path
content_policy_fallbacks	ContentPolicyViolationError

This split matters. If a prompt is too long, retrying the same model is pointless. You want a bigger context window, not a second try. In my simulation, a context error went straight to the larger model, and the ordinary backup was never called.

So, Are Auth Errors Retried or Not?

I could not confirm the default. The Router page does not state it. Here is what the pages do say [2][3]:

1. RetryPolicy has an AuthenticationErrorRetries field, and the docs' example sets it to 0.
2. The cooldown table on the Router page lists "401 (Auth)" as non-retryable.
3. The reliability page never mentions authentication errors, so it is unclear whether they go to the catch-all fallbacks list.

The safe move is to set AuthenticationErrorRetries to 0 explicitly and add a fallback, so you do not depend on an undocumented default.

A Practical Setup

Put the pieces together in one config:

model_list:
  - model_name: primary
    litellm_params:
      model: openai/<primary-model>
  - model_name: backup
    litellm_params:
      model: anthropic/<backup-model>
  - model_name: long-context
    litellm_params:
      model: anthropic/<long-context-model>

router_settings:
  num_retries: 2
  allowed_fails: 3
  cooldown_time: 30
  retry_policy:
    AuthenticationErrorRetries: 0
    RateLimitErrorRetries: 3
    ServiceUnavailableErrorRetries: 2
    NotFoundErrorRetries: 0
    DefaultRetries: 1

litellm_settings:
  fallbacks: [{"primary": ["backup"]}]
  context_window_fallbacks: [{"primary": ["long-context"]}]
  content_policy_fallbacks: [{"primary": ["backup"]}]

Then hold yourself to four rules:

1. Set retry_policy per error type. Use 0 for authentication, a few retries for rate limits, and short counts for 503.
2. Put a fallback model group on fallbacks, and a larger-context model on context_window_fallbacks.
3. Set cooldown_time and allowed_fails deliberately instead of trusting the defaults.
4. Alert on 401s. A fallback that quietly hides a dead key is how you end up with a surprise bill and no error.

Test each path with a mock failure. Note that from proxy v1.85.0 the mock_testing_* flags are ignored on proxy requests and work only with direct litellm.Router calls [3].

Common Pitfalls

1. Retrying a 401. The key will not heal. Set the retry count to 0 and alert instead.
2. Trusting the documented cooldown duration. "A minute" and 5s both appear on the same docs site.
3. Leaving DefaultRetries alone and expecting 404s to surface. Unset fields fall through, so a 404 is retried unless you say otherwise.
4. Relying on catch-all fallbacks for auth failures. The docs do not confirm that path; add an explicit fallback or let it fail loudly.
5. Forgetting context_window_fallbacks. A too-long prompt retried on the same model burns the whole retry budget and still fails.
6. Setting per-deployment limits in litellm_params. Those values get sent to the provider. Use model_info.
7. Testing fallbacks with mock_testing_* through the proxy. Since v1.85.0 those flags are stripped, so you will see nothing happen.
8. Letting callers raise x-litellm-num-retries uncontrolled. On the proxy, that header outranks your config.

Wrapping Up

Retries fix transient problems, fallbacks fix broken deployments, and cooldowns stop you from hammering something that is already down. The trick is matching each error to the right tool: zero retries for bad keys, a few for rate limits, a bigger model for long prompts.

Pull up your current config today and check what it does with a 401. If the answer is "retry three times", you now know what that costs.

If you are still setting up the gateway, the LiteLLM proxy guide covers the quickstart, and the budgets guide covers the spend side of the same config file.

Frequently Asked Questions

Should LiteLLM retry a 401 authentication error?

No. A wrong or expired key does not fix itself between attempts, so retrying spends calls and time for nothing. Set AuthenticationErrorRetries to 0 in the retry policy and route the failure to a fallback model group, or let it surface so someone rotates the key. The docs' own retry_policy example sets AuthenticationErrorRetries to 0, and the cooldown table lists 401 as non-retryable.

What are the default allowed_fails and cooldown_time in LiteLLM?

The documented defaults are allowed_fails 3 and cooldown_time 5 seconds, the latter defined as DEFAULT_COOLDOWN_TIME_SECONDS in constants.py. The docs are inconsistent about the duration elsewhere, describing a model as cooled down for a minute while the default is five seconds, so set both values explicitly instead of trusting either statement.

Do auth errors go to the general fallbacks list?

The docs do not say clearly. The reliability page never mentions authentication errors, so it is unclear whether a 401 reaches the catch-all fallbacks list after retries. The cooldown table does list 401 as non-retryable, and the retry policy exposes AuthenticationErrorRetries, which suggests the failure either falls back immediately or surfaces. Set AuthenticationErrorRetries to 0 and add an explicit fallback so you do not depend on an undocumented default.

What is the difference between fallbacks, context_window_fallbacks and content_policy_fallbacks?

fallbacks covers all remaining errors such as rate limits and 500s. context_window_fallbacks applies to ContextWindowExceededError, so an over-long prompt goes to a model with a bigger window instead of being retried against the same limits. content_policy_fallbacks applies to ContentPolicyViolationError. Adding the specific lists matters because retrying a too-long prompt on the same model can never succeed.

Where does num_retries come from when several places set it?

Four places, ranked highest first: the x-litellm-num-retries request header, which only the proxy honours; num_retries in the request body; num_retries in a deployment's litellm_params; and num_retries in litellm_settings as the router-wide default. A caller can therefore always raise or lower the setting for one request, which is useful for long batch jobs and dangerous if you have not capped the header at the edge.

How do I test fallbacks when the mock_testing flags no longer work?

Since Proxy v1.85.0 the mock_testing_fallbacks, mock_testing_context_fallbacks and mock_testing_content_policy_fallbacks flags are stripped from incoming proxy requests and have no effect. They still work for direct litellm.Router calls in tests. To exercise the paths end to end through the proxy, point one deployment at a deliberately bad key or a tiny-context model and watch which model actually answers.

Sources

• [1] LiteLLM, Exception mapping — the provider-to-OpenAI exception table including AuthenticationError, PaymentRequiredError with its Router cooldown note, ContextWindowExceededError and ContentPolicyViolationError, and the litellm._should_retry(e.status_code) example with no documented return values: docs.litellm.ai/docs/exception_mapping
• [2] LiteLLM, Router — num_retries and its four-level precedence, the RetryPolicy fields and the AuthenticationErrorRetries: 0 example, the unset-field fallthrough that retries 404s, allowed_fails and cooldown_time defaults of 3 and 5s (DEFAULT_COOLDOWN_TIME_SECONDS), the per-deployment model_info caveat, allowed_fails_policy, and the cooldown table listing 401 as non-retryable: docs.litellm.ai/docs/routing
• [3] LiteLLM, Fallbacks — the "fails after num_retries" definition, the three fallback lists, the allowed_fails: 3 comment about failing more than one call in a minute, the absence of any authentication-error section, and the note that mock_testing_* flags are stripped from proxy requests from v1.85.0: docs.litellm.ai/docs/proxy/reliability

Related Articles

• LiteLLM Proxy Server: Complete Guide to Running One API for Every LLM Provider
• LiteLLM Virtual Keys and Budgets: Control Who Spends What
• How to Share One LiteLLM API Endpoint Across Your Team