LiteLLM Timeouts: The Setting Your Load Balancer Kills

Contents

Figure 1: LiteLLM Timeouts: The Setting Your Load Balancer Kills — two hops, two sets of deadlines, one visible failure

A LiteLLM proxy looks simple on a diagram: apps on one side, model providers on the other. In production, it is two separate HTTP hops with different failure modes. Your app talks to the proxy, and the proxy talks to OpenAI, Anthropic, Azure, or whoever else you route to. A load balancer usually sits in front of the proxy, too.

Each hop has its own timeouts, keepalives and connection pools, and the defaults rarely match your workload. Long reasoning requests die at a load balancer's idle timeout. A hung provider holds a request for ten minutes. A traffic spike saturates one worker's event loop and Kubernetes restarts the pod.

This guide covers the HTTP settings that fix those problems: timeouts, keepalives, connection tuning and the somewhat confusing topic of compression. I read LiteLLM's own documentation pages, which showed update dates of 27 and 28 October 2026, but this project changes constantly. Check the docs for your exact version before you apply anything, and load test before you roll it out.

Two Hops, Two Sets of Settings

Hop	What you are tuning
Client to proxy	Server keepalive timeout, workers, TLS, HTTP/2 serving, admission control and SSE keepalive pings [2]
Proxy to provider	Request and stream timeouts, retries, transport choice (aiohttp or httpx), connection pool limits, outbound HTTP/2, SSL and proxy variables [1][2][3]

Most surprises come from tuning one hop and forgetting the other.

Timeouts

LiteLLM offers timeouts at several levels [1]:

litellm_settings.request_timeout, or the REQUEST_TIMEOUT environment variable, sets the proxy-wide default when nothing more specific applies. The server tuning docs say the request timeout defaults to 600 seconds [2].
router_settings.timeout sets a timeout for the entire call across the router [1].
Per deployment, under litellm_params: timeout caps the complete response, and stream_timeout caps the wait for the first chunk of a streaming response [1].
Per request: a timeout parameter in the request body, or the x-litellm-timeout header [1].
Per service tier: timeout_by_service_tier and stream_timeout_by_service_tier give requests that ask for a tier, like OpenAI's slower flex tier, their own limits [1].

One docs inconsistency worth knowing: the server tuning page says the request timeout defaults to 600 seconds, while the production page states 6000 [2]. If your version's behaviour matters, test it rather than trusting either number.

Precedence

For each attempt, LiteLLM picks the first value that is set [1]:

A timeout sent with the request
The service-tier timeout
The deployment's timeout
The router and global defaults

Streaming requests follow the same order with the stream variants [1].

Timeout Versus Stream Timeout

These two catch different failures:

timeout protects you from calls that run too long overall.
stream_timeout protects you from providers that accept the request and then say nothing. The docs suggest it for aborting hanging providers so the router can retry another deployment [1].

Two details deserve attention.

Some requests cannot stream from the provider, like a JSON-schema response format on a model without native structured output. LiteLLM makes one non-streaming call and sends the answer as chunks afterward, and these requests use the longer of timeout and stream_timeout, so a short stream timeout does not cut off a slow structured answer [1].

On direct SDK calls without a router, stream_timeout now applies too. Earlier versions ignored it, so a short value you set there can now cut off a live stream that is slow to send a chunk [1].

The proxy-wide request_timeout also applies to the native /v1/responses and /v1/messages streams, where it bounds each wait for the next chunk, so a stalled upstream ends the stream with an error instead of hanging [1].

A Sensible Starting Config

The values below are illustrative. The docs' own examples use tiny timeouts like 0.1 seconds to demonstrate behaviour, and you do not want those in production.

litellm_settings:
  request_timeout: 600            # proxy-wide default, in seconds

router_settings:
  timeout: 600                    # whole-call timeout at the router

model_list:
  - model_name: chat-main
    litellm_params:
      model: azure/my-deployment-eu
      api_base: https://my-endpoint-eu.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY
      timeout: 300                # complete response cap
      stream_timeout: 30          # wait for first chunk
      max_retries: 2
  - model_name: chat-main
    litellm_params:
      model: azure/my-deployment-ca
      api_base: https://my-endpoint-ca.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY
      timeout: 300
      stream_timeout: 30
      max_retries: 2

With two deployments behind one model name, a deployment that hangs past stream_timeout lets the router try the other one. Size stream_timeout from real time-to-first-token measurements, with headroom. Reasoning models can legitimately stay silent for a long time before the first token, so a 30-second limit that works for a small chat model may break a thinking model.

Per-Tier Timeouts

If you use a provider tier that runs slower by design, give it its own limit instead of raising the default for everyone [1]:

model_list:
  - model_name: chat-main
    litellm_params:
      model: openai/my-model
      api_key: os.environ/OPENAI_API_KEY
      timeout: 610
      stream_timeout: 610
      timeout_by_service_tier:
        flex: 900
      stream_timeout_by_service_tier:
        flex: 900

A request without a service_tier, or with a tier that has no entry, keeps the deployment's normal limits. Tier names are not validated against the provider, so a misspelled key never matches [1].

Per-Request Timeouts and Testing

Clients can pass timeout in the request body, including through extra_body with the OpenAI SDK [1]. To test your retry and fallback logic, the docs describe a mock_timeout: true flag, supported on the chat completions and text completions endpoints [1]. Use it in staging to prove that a timeout really triggers a fallback.

Keepalive Pings for Long Silent Streams

Timeouts limit how long a request may run. A separate problem hits streaming requests with long silent gaps, like extended thinking models: load balancers and reverse proxies close connections that look idle, even though a client is legitimately waiting [1].

LiteLLM's fix is an SSE comment frame. After a stream stays silent for longer than the configured interval, the proxy sends : ping down the connection, repeating until real content resumes. SSE clients and intermediaries are expected to ignore comment frames [1].

# Per deployment
model_list:
  - model_name: reasoning-model
    litellm_params:
      model: anthropic/my-reasoning-model
      api_key: os.environ/ANTHROPIC_API_KEY
      keepalive_seconds: 15

# Or one default for every deployment and pass-through route
litellm_settings:
  sse_keepalive_ping_interval_seconds: 15

Details that matter [1]:

Behaviour	What the docs say
Clamping	The effective value is clamped to 1-300 seconds
Hard disable	A deployment-level `keepalive_seconds: 0` overrides everything, including a client override grant
Client override	A request-level `keepalive_seconds` or the `x-litellm-keepalive-seconds` header is silently ignored unless the deployment sets `allow_client_keepalive_override: true`
Why clients cannot enable it	Any client could keep an idle-looking stream alive past a load balancer's timeout and tie up a `max_parallel_requests` slot
Global-only side effect	The global setting covers silence before the upstream answers: the proxy opens the SSE response early, pings, then replays the real response
Cost of that	*The 200 is committed before the outcome is known, so a later failure arrives as an SSE error frame under a 200, and the `x-litellm-` response headers are absent on a stream that pinged**

Pick a ping interval comfortably shorter than the shortest idle timeout in front of the proxy, whether that is a cloud load balancer, an ingress controller or a CDN. That number is yours to check, since it lives outside LiteLLM.

Compression: Two Very Different Things

"Compression" means two things in LLM gateways, and the confusion wastes time.

HTTP-Level Compression (gzip)

I did not find a dedicated HTTP compression setting in the LiteLLM pages I read. What I did find: a community discussion proposing a flag to disable gzip on upstream requests by sending Accept-Encoding: identity, where a maintainer asked for a performance measurement before turning it into a feature request. Treat that as an idea, not a shipped feature.

From general practice, not from LiteLLM's docs: compress at the edge, like your ingress or CDN, if you need it for large client payloads. And be careful with streaming. Compression layers can buffer small SSE chunks and delay tokens, so test streaming through any compressing layer before you enable it. LLM traffic is mostly small JSON and event streams, so gzip rarely changes much, and the CPU cost deserves a measurement.

Prompt and Context Compression

In LiteLLM's documentation, "compression" mostly means shrinking prompts before they reach the model, which reduces tokens, cost and latency. Two guardrail integrations appear in the docs [5]:

Integration	Requirement	Endpoint
Headroom	LiteLLM v1.92.0 or later, plus a reachable Headroom proxy	Messages go to `{api_base}/v1/compress`; the returned list replaces the payload
Compresr	A LiteLLM build that includes the compresr guardrail plus an API key	`https://api.compresr.ai` by default, model `latte_v2`

Both follow the same pattern [5]:

Opt in per request with a guardrails array, per key, or for everyone with default_on: true. The Compresr docs recommend leaving the default off so compression stays opt-in.
Confirm it ran by checking the x-litellm-applied-guardrails response header.
System prompts stay untouched by default, via compress_system: false.

Three operational details the marketing pages skip [5]:

Headroom's guardrail is mode: pre_call, and it is "a no-op on responses" — only the request path is rewritten.
Compresr compresses tool and function outputs of at least 500 characters by default; system messages, prior history and the last user message pass through unless you opt them in.
Compresr's unreachable_fallback defaults to fail_closed, so an unreachable compression service fails the request unless you change it.

Before you enable either one, weigh the trade-offs: another network dependency, extra latency, a third party that sees your prompts, and possible quality loss from dropped context. Test answer quality on your own evaluation set.

Connection Tuning

Pick the Transport

LiteLLM's outbound calls use aiohttp by default now. The benchmark page says aiohttp was opt-in at v1.71.1 and is now the default with no setting needed. To get httpx instead, set DISABLE_AIOHTTP_TRANSPORT="True" or litellm.disable_aiohttp_transport = True, and enabling HTTP/2 also forces httpx [4].

Why it changed, in the docs' own numbers: on a 2 vCPU, 4GB test machine at 200 RPS, aiohttp hit 224 RPS against httpx's 50.2, with median latency falling from 2,500ms to 74ms [4].

Outbound HTTP/2

From LiteLLM v1.103.0 on, you can negotiate HTTP/2 with providers [2]:

litellm_settings:
  http2: true      # or the LITELLM_HTTP2 environment variable

The docs list caveats [2]:

It routes provider traffic through httpx instead of aiohttp, so you give up aiohttp's HTTP/1.1 throughput advantage. Load test before enabling it fleet-wide.
Upstreams that do not offer h2 fall back to HTTP/1.1, and plain http:// upstreams stay on HTTP/1.1.
Clients you pass in yourself through litellm.client_session or litellm.aclient_session are not switched.
Deployments on the aiohttp_openai/ provider always use aiohttp, and LiteLLM logs a warning if the flag is on.

These flags affect only the proxy-to-provider hop. Serving HTTP/2 to your own clients is a separate step with Hypercorn and --run_hypercorn [2].

Connection Pool Variables

A third-party code analysis lists environment variables for the aiohttp connector: AIOHTTP_CONNECTOR_LIMIT, AIOHTTP_CONNECTOR_LIMIT_PER_HOST, AIOHTTP_KEEPALIVE_TIMEOUT and AIOHTTP_TTL_DNS_CACHE. I did not find these on the official pages I opened, so check your version's documentation or source before relying on them.

Why they matter, from general networking practice: a connection limit that is too low makes requests queue inside the proxy under high concurrency, and a keepalive timeout longer than a provider's idle timeout produces errors when the proxy reuses a connection the provider already closed. Raise limits deliberately, and watch for connection resets.

Client-to-Proxy Keepalive

The server tuning docs say the proxy's keepalive timeout defaults to 5 seconds: connections must receive new data within that period or get disconnected. Change it with --keepalive_timeout or the KEEPALIVE_TIMEOUT environment variable; the example uses 75 [2].

A rule from general practice, not from the docs: set it longer than your load balancer's idle timeout, so the proxy does not close a connection that the balancer still thinks it can reuse.

Workers and Scaling

The docs recommend one Uvicorn worker per pod and horizontal scaling, because that keeps latency predictable and makes rolling restarts hitless [2]. Reach for Gunicorn only if you must pack several workers into one container [2].

Two cautions [2]:

Use --max_requests_before_restart only with two or more workers. With one worker, hitting the limit takes the whole pod out of service. Health probes count toward the limit, so two probes every 15 seconds use up 11,520 requests a day.
Add --max_requests_before_restart_jitter so workers do not all recycle at once.

A Granian server option exists in beta. LiteLLM's own benchmarks report a 10 to 20 requests-per-second improvement over Uvicorn at the same worker count, which is self-reported, so test it yourself [2]. Granian does not support --max_requests_before_restart or --ciphers, and it requires both TLS flags when you enable TLS [2].

Hitless Restarts

Long LLM requests make deploys risky. The docs recommend a rolling update strategy, a readiness probe on /health/readiness, a terminationGracePeriodSeconds comfortably larger than your longest request, and an optional short preStop sleep [2]:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 620   # longer than request_timeout: 600
      containers:
        - name: litellm
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 4000
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]

Admission Control: Shed Load Before the Event Loop Drowns

This setting solves a nasty failure mode. A saturated worker keeps accepting connections, callers wait seconds with no overload signal, the liveness probe slows down, and Kubernetes restarts the pod, which dumps its load on the remaining replicas [2].

Admission control caps the work per worker and turns the excess into a fast, retryable 503 [2]:

general_settings:
  max_in_flight_requests_per_worker: 64
  max_queued_requests_per_worker: 64
  admission_queue_timeout_seconds: 1.0

A rejected request gets 503 with retry-after: 1 and an overloaded_error body naming the in-flight and queued counts [2]. The details that change how you size it [2]:

It is off until you set the in-flight limit, and the limits are read on the first request, so changing them needs a restart.
max_queued_requests_per_worker defaults to the in-flight cap.
A streaming completion holds its slot until the last chunk, and a client disconnect frees the slot immediately.
Health and metrics paths bypass the gate, so a busy worker still answers its liveness probe. So does /health/backlog.
Rejections happen before authentication, so they are not attributed to a key in spend logs.
Limits apply per worker process: four workers at 64 admit up to 256 concurrent requests, and each replica multiplies that.
Monitor it with /health/backlog or the litellm_admission_* Prometheus metrics. Steady queue_timeout rejections mean you need more replicas; queue_full rejections mean spikes outrun the queue.
It complements global_max_parallel_requests, which is a deployment-wide limit coordinated through Redis: use the global limit to protect providers and the per-worker limit to protect each event loop.

Your clients must handle 503 with retry-after gracefully, so make sure your SDK retry settings fit.

SSL, Proxies and IPv4

The security settings docs describe several network knobs [3]:

Setting	What it does
`ssl_verify`	A custom CA bundle path (or `SSL_CERT_FILE`), or `False` to disable verification
`ssl_certificate`	A client-side certificate for mutual TLS
`ssl_security_level`	Cipher strength, for example `DEFAULT@SECLEVEL=1`
`ssl_ecdh_curve`	The ECDH curve; `X25519` "disables PQC for better performance"
`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`	Read for every LLM request on both transports
`DISABLE_AIOHTTP_TRUST_ENV`	Makes the aiohttp transport ignore those proxy variables
`force_ipv4`	Forces IPv4 for all LLM requests

Environment variables take precedence over SDK settings [3]. The ssl_ecdh_curve note is a security trade-off, not a free win: setting X25519 turns off post-quantum key exchange to speed up handshakes, so decide it deliberately [3]. An invalid curve name logs a warning and falls back to default curves [3].

Symptom Cheat Sheet

Symptom	First move
Streams die after a fixed idle time, with load balancer errors	Add SSE keepalive pings shorter than the balancer's idle timeout
A provider hangs and requests pile up	Set `stream_timeout` per deployment, add a second deployment or fallback, test with `mock_timeout`
Slow tiers get cut off	Add `timeout_by_service_tier` entries
Pods restart during traffic spikes	Add admission control, and scale replicas
Memory grows slowly	Recycle workers, with two or more workers and jitter
Deploys cut off long requests	Raise `terminationGracePeriodSeconds`, use readiness probes and rolling updates
Connection resets on reused connections	Compare keepalive settings on both hops against provider and balancer idle timeouts
Throughput plateaus under high concurrency	Check connector limits, worker counts and replica counts

Test Before You Ship

Pin your LiteLLM version and read that version's docs.
Load test with a realistic mix of short and long requests, including streaming.
Prove the timeouts: use mock_timeout to confirm that retries and fallbacks fire [1].
Watch pings: run a streaming request with curl -N and check that comment frames arrive during silence.
Break a provider in staging, and watch the router's behaviour.
Watch /health/backlog and your metrics during the test.
Compare transports: if you consider HTTP/2, test it against the default.

Common Pitfalls

Setting one timeout for everything. Chat models, reasoning models and batch jobs need different limits.
Forgetting the load balancer. Its idle timeout often kills long streams first.
Copying doc examples literally. Some use tiny timeouts to demonstrate behaviour.
Using worker recycling with one worker. The pod goes out of service.
Enabling HTTP/2 everywhere without testing. It changes the transport.
Mixing up the two kinds of compression. Prompt compression and HTTP compression solve different problems.
Skipping retry handling for 503s. Admission control assumes clients retry.
Leaving unreachable_fallback at its default on Compresr. The default is fail-closed.
Trusting either default for request_timeout. Two docs pages disagree.

Best Book to Read

Cover	Book	Description	Get it
	Systems Performance: Enterprise and the Cloudby Brendan Gregg	latency, queues and CPU measurement done properly, which is how you decide whether any of these settings helped.	View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Wrapping Up

LiteLLM's HTTP settings come down to a few ideas: give every request a sensible deadline, keep idle-looking streams alive, size connections and workers for your traffic, and shed load before a worker drowns. The defaults serve small deployments well, and production traffic will find every gap.

Start with timeouts and keepalives, since they cause the most visible failures. Then load test your changes, one at a time, against the exact LiteLLM version you run.

If you have not deployed the proxy yet, the Docker production guide covers the container, keys and probes. The load balancing guide covers what happens when a timeout sends traffic to a second deployment, and the streaming guide covers the client side of keepalive pings.

Frequently Asked Questions

Which timeout setting wins in LiteLLM?

The first one that is set, in this order: a timeout sent with the request, then the service-tier timeout, then the deployment's timeout under litellm_params, then the router and global defaults. Streaming requests follow the same order with the stream variants. litellm_settings.request_timeout or the REQUEST_TIMEOUT environment variable is the proxy-wide fallback when no router or deployment timeout applies.

What is the difference between timeout and stream_timeout in LiteLLM?

timeout caps the complete response, while stream_timeout caps the wait for the first chunk of a streaming response. timeout protects you from calls that run too long overall; stream_timeout protects you from providers that accept the request and then say nothing, so the router can retry another deployment. Requests the provider cannot stream, such as a json_schema response format without native structured output, use the longer of the two.

Why do my streaming responses die after a fixed idle period?

Because a load balancer, ingress or CDN in front of the proxy closed a connection that looked idle. LiteLLM can send SSE comment frames during silent gaps: set keepalive_seconds per deployment or sse_keepalive_ping_interval_seconds globally, and pick an interval comfortably shorter than the shortest idle timeout in front of the proxy. The effective value is clamped to 1-300 seconds, and a deployment-level keepalive_seconds: 0 is a hard disable that nothing can override.

What is max_in_flight_requests_per_worker in LiteLLM?

Per-worker admission control. It caps how much work each worker process accepts and turns the excess into a fast 503 with an overloaded_error body and a retry-after header, instead of letting latency pile up until the liveness probe fails and Kubernetes restarts the pod. It is off until you set the in-flight limit, limits are read on the first request so changes need a restart, and health and metrics paths bypass the gate.

Does LiteLLM compress requests with gzip?

I did not find a dedicated HTTP compression setting on the LiteLLM pages I read. What the docs call compression is prompt compression: guardrail integrations such as Headroom and Compresr that rewrite messages before the call to save tokens. For HTTP-level compression, compress at your ingress or CDN if you need it, and test streaming through that layer because compressing intermediaries can buffer small SSE chunks.

Is aiohttp or httpx the default transport in LiteLLM?

aiohttp. It became the default after benchmarks showed 224 RPS against 50.2 RPS for httpx on the same 2 vCPU test machine, with median latency falling from 2,500ms to 74ms. Set DISABLE_AIOHTTP_TRANSPORT=True or litellm.disable_aiohttp_transport = True to get httpx back, and note that enabling outbound HTTP/2 also forces httpx.

Sources

[1] LiteLLM, Timeouts — the litellm_settings.request_timeout and REQUEST_TIMEOUT proxy-wide default with its application to /v1/responses and /v1/messages streams, router_settings.timeout, per-deployment timeout and stream_timeout, the request-body timeout and x-litellm-timeout header, timeout_by_service_tier and stream_timeout_by_service_tier, the four-step precedence order, the non-streamable request rule using the longer of the two timeouts, stream_timeout applying to direct SDK calls, keepalive_seconds with allow_client_keepalive_override, the x-litellm-keepalive-seconds header, the 1-300 second clamp, the hard disable at 0, the early-SSE-response side effect, and mock_timeout limited to chat and text completions: docs.litellm.ai/docs/proxy/timeout
[2] LiteLLM, Server tuning — one Uvicorn worker per pod with horizontal scaling, Gunicorn for multiple workers per container, --max_requests_before_restart with its single-worker warning and the 11,520 requests a day from health probes, --max_requests_before_restart_jitter, the hitless rolling-update snippet with terminationGracePeriodSeconds: 620 and the preStop sleep, TLS at the proxy, Hypercorn for serving HTTP/2, outbound HTTP/2 since v1.103.0 with its httpx and aiohttp_openai/ caveats, Granian beta with its 10-20 RPS claim and unsupported flags, per-worker admission control with max_in_flight_requests_per_worker, max_queued_requests_per_worker, admission_queue_timeout_seconds, the overloaded_error 503 body, the bypassed health paths, /health/backlog and the Redis-backed global_max_parallel_requests, and the 5-second keepalive default with a 75 example: docs.litellm.ai/docs/proxy/server_tuning
[3] LiteLLM, Security settings — ssl_verify with a custom CA bundle or SSL_CERT_FILE, ssl_verify: false, ssl_security_level, ssl_certificate for mutual TLS, ssl_ecdh_curve: X25519 disabling PQC for performance with a warning-and-fallback on invalid names, HTTP_PROXY/HTTPS_PROXY/NO_PROXY applying on both transports with or without force_ipv4, and DISABLE_AIOHTTP_TRUST_ENV: docs.litellm.ai/docs/guides/security_settings
[4] LiteLLM, aiohttp benchmarks — USE_AIOHTTP_TRANSPORT as the opt-in flag that no longer exists, aiohttp as the default transport, DISABLE_AIOHTTP_TRANSPORT and litellm.disable_aiohttp_transport to get httpx back, HTTP/2 forcing httpx, and the 200 RPS comparison on 2 vCPU/4GB showing 50.2 versus 224 RPS with median latency of 2,500ms versus 74ms: docs.litellm.ai/docs/aiohttp_benchmarks
[5] LiteLLM, Headroom and Compresr guardrails — Headroom requiring v1.92.0 and a reachable proxy, posting to {api_base}/v1/compress and replacing the payload, mode: pre_call being the only meaningful mode and a no-op on responses, the optional api_key, default_on, the x-litellm-applied-guardrails header, and x-headroom-bypass only ever switching compression off; plus Compresr's required API key or COMPRESR_API_KEY, default api_base and latte_v2 model, unreachable_fallback defaulting to fail_closed, the 500-character tool-output threshold with compress_system, compress_history and compress_last_user opt-ins, and tokens_saved in spend logs: docs.litellm.ai/docs/proxy/headroom and docs.litellm.ai/docs/proxy/guardrails/compresr