LiteLLM Streaming: The Chunk That Breaks Your Loop

Contents

Figure 1: LiteLLM Streaming: The Chunk That Breaks Your Loop — the text arrives in pieces, and so do the errors

Streaming is the difference between a chat that feels alive and one that stares at you for eight seconds. LiteLLM makes it a single argument. The trouble starts when you want token counts, async code or sane failure handling, because those details sit on different docs pages.

LiteLLM is not installed here and PyPI is unreachable, so I never ran its code. The snippets below are adapted from LiteLLM's docs and untested by me. What I did run is a streaming client against a local mock server, to test the failure cases any LiteLLM user will meet.

The Basics: stream=True

The docs' sync example passes stream=True and loops over the response [1]:

from litellm import completion

response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
for chunk in response:
    print(chunk["choices"][0]["delta"].content or "")

Each chunk carries choices[0].delta, and the text is in delta.content. That is the only chunk detail the main streaming page shows, so do not assume more fields without checking your provider [1].

Async Streaming

The async version uses acompletion with stream=True. Per the docs, the streaming object implements __anext__(), "which enables async iteration" [1]:

import asyncio
from litellm import acompletion

async def main():
    response = await acompletion(model="gpt-3.5-turbo", messages=messages, stream=True)
    async for chunk in response:
        print(chunk.choices[0].delta.content or "", end="")

asyncio.run(main())

A caution: the page's plain async example — the one that returns a whole response — does not pass stream=True, and the page never shows stream_chunk_builder with an async stream [1]. Test that combination yourself before you rely on it.

Rebuilding the Full Response

Sometimes you want the complete message after streaming. LiteLLM "exposes a helper function to rebuild the complete streaming response from the list of chunks" [1]. The docs' example ends with litellm.stream_chunk_builder(chunks, messages=messages). Their snippet leaves out the import litellm line and the chunks = [] setup, so a working version looks like this:

import litellm

chunks = []
for chunk in litellm.completion(model="gpt-3.5-turbo", messages=messages, stream=True):
    chunks.append(chunk)
full = litellm.stream_chunk_builder(chunks, messages=messages)

My own client does the same job with a simplified rebuild: join the delta text, keep the finish reason and grab the usage chunk. In my test, six chunks became "Hello world" with a stop reason and nine total tokens.

Token Usage in Streams

Streams do not report usage unless you ask. You ask with stream_options={"include_usage": True}, the OpenAI-compatible option LiteLLM forwards. On the proxy there is a documented switch for it: always_include_stream_usage under general_settings defaults to false, and setting it true "includes usage metrics in every streaming response chunk" [3]. The proxy applies it to every deployment without changing your other stream options.

Here is what changes in the stream:

1. An extra chunk arrives before the [DONE] message.
2. That chunk's usage field holds totals for the whole request.
3. Its choices array is empty.
4. Every other chunk has usage set to null.

Point 3 is the classic trap. The docs' own loop guards with if chunk.choices: before touching chunk.choices[0], and prints chunk.usage in the else branch [1]. In my test, the unguarded loop raised IndexError on the final chunk, while the guarded one worked.

for chunk in response:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    elif chunk.usage:
        print("\n", chunk.usage)

One caveat I cannot resolve from the docs: providers differ, so confirm the usage chunk actually arrives for the model you stream. A missing usage chunk looks exactly like a stream that worked.

When Streams Fail

The main streaming page covers one failure case: a model repeating the same chunk. LiteLLM checks for it, and past the limit it raises litellm.InternalServerError. The limit is litellm.REPEATED_STREAMING_CHUNK_LIMIT = 100 in the SDK, or REPEATED_STREAMING_CHUNK_LIMIT: 100 under litellm_settings on the proxy [1]. The page says this "allows retry logic to happen" [1].

What it does not hand you is a plain exception to catch, but it does hand you a reproduction. The docs build a ModelResponse list of loop_amount = litellm.REPEATED_STREAMING_CHUNK_LIMIT + 1 identical chunks, wrap it in ModelResponseListIterator, pass it to CustomStreamWrapper, and iterate — which raises without touching a provider [1]. That is the cheapest way to prove your except litellm.InternalServerError path actually runs.

The page says nothing about dropped connections or provider errors mid-stream. Those are on you. My client test covers three cases:

Case	What my client did
Dropped connection	The server sent two chunks and then died with no [DONE]. I kept the partial text ("partial answer") and raised an error, because a stream that ends without [DONE] is a failure.
Silent stream	With a 0.5-second first-chunk limit, a server that slept for two seconds triggered a stall error after 0.50 seconds
Runaway repeats	A server that looped one chunk forever was stopped by my own guard at 20 repeats. Your real limit should come from the docs' setting, not my demo number.

Timeouts and Keepalives

From LiteLLM's timeout page, per model under litellm_params [2]:

Setting	What it caps
timeout	The complete response (the docs' example is 300)
stream_timeout	The wait for the first chunk (the docs' example is 30)
keepalive_seconds	Sends SSE : ping comments during silent gaps, clamped to 1-300 seconds, with 0 disabling it

On direct SDK calls stream_timeout applies to completion(), acompletion() and aresponses(); earlier versions ignored it there [2].

The proxy adds a global default the deployment-level pages do not emphasise: sse_keepalive_ping_interval_seconds applies to every deployment that does not set its own keepalive_seconds, and to pass-through routes, which have no deployment of their own [3].

Ping lines start with a colon and are not data, so your parser must skip them. In my test, a stream with two pings and one real chunk produced exactly one chunk. The docs do not state a default for stream_timeout or keepalive_seconds [2].

A Streaming Checklist

1. Request usage with stream_options={"include_usage": True}, or set always_include_stream_usage on the proxy.
2. Guard every loop with if chunk.choices:.
3. Treat a missing [DONE] as an error, and keep the partial text.
4. Set stream_timeout so a silent model cannot hang a user.
5. Skip SSE comment lines in custom clients.
6. Catch InternalServerError from the repeated-chunk guard and decide whether to retry.
7. Test stream_chunk_builder with your own async code path.

Common Pitfalls

1. Indexing choices[0] without a guard. The usage chunk has none.
2. Assuming every chunk has text. The last ones do not.
3. Forgetting stream_options and wondering where the token counts went. Nothing is reported by default.
4. Parsing SSE comments as data. : ping is not a chunk.
5. Treating a stream that ended early as complete. Without [DONE] you have a fragment.
6. Setting keepalive_seconds above 300 or below 1. It is clamped, not rejected.
7. Copying the docs' helper snippet verbatim. It is missing the import and the list.
8. Catching nothing when the repeat guard fires. InternalServerError is the retry signal.

Wrapping Up

Streaming in LiteLLM is easy to turn on and easy to get subtly wrong. The usage chunk has empty choices, failures arrive as half-finished answers and the docs only cover one stream failure by name.

Guard your loops, set a stream timeout and test a dropped connection before your users do.

If retries and fallbacks are new to you, the retries and fallbacks guide covers which errors deserve another attempt, and the error triage guide covers the status codes that hide behind them. For the deployment side of streaming limits, see the load balancing guide.

Frequently Asked Questions

How do I stream a response with LiteLLM in Python?

Pass stream=True to completion() and iterate over the returned object, reading each chunk at chunk.choices[0].delta.content. The async form is acompletion() with stream=True in an async for loop, because the streaming object implements anext(). Always guard the loop with if chunk.choices, since the usage chunk and the final chunk carry an empty choices array.

Why does my LiteLLM streaming loop raise IndexError on the last chunk?

Because the final chunks are not text chunks. When you request usage with stream_options include_usage, an extra chunk arrives before the end of the stream whose choices array is empty and whose usage field holds the totals for the whole request; other chunks carry usage set to null. Indexing chunk.choices[0] on that chunk raises IndexError, so check if chunk.choices first and read chunk.usage in the else branch.

Does LiteLLM report token usage for streamed requests?

Only when you ask. You request it with stream_options and include_usage, the OpenAI-compatible option LiteLLM forwards. On the proxy you can set always_include_stream_usage to true under general_settings, which the config reference describes as including usage metrics in every streaming response chunk, with a default of false. Providers differ, so verify the usage chunk exists for the model you are actually streaming.

What is stream_chunk_builder in LiteLLM?

It is a helper that rebuilds a complete response from a list of streamed chunks, called as litellm.stream_chunk_builder(chunks, messages=messages). The docs snippet omits both the litellm import and the chunks list initialisation, so add those yourself. The page shows it only for a synchronous stream, so test the async path on your own code before relying on it.

How does LiteLLM handle a model that repeats the same chunk?

It raises litellm.InternalServerError once a chunk repeats more than litellm.REPEATED_STREAMING_CHUNK_LIMIT times, which defaults to 100 on the SDK and can be set under litellm_settings on the proxy. The point of raising an error rather than stopping quietly is to let retry logic run, so pair it with retries or a fallback deployment. The docs also publish a reproduction snippet that raises it without calling a provider.

How do I stop a stream that goes silent?

Set a stream timeout. On a deployment, stream_timeout caps the wait for the first chunk, while timeout caps the complete response. For long silences inside a stream, keepalive_seconds sends SSE comment lines that are clamped to between 1 and 300 seconds, with 0 disabling them, and the proxy has a sse_keepalive_ping_interval_seconds default that applies to every deployment without its own value.

Sources

• [1] LiteLLM, Streaming + Async — stream=True with the part.choices[0].delta.content or "" loop, the stream_chunk_builder helper shown without import litellm or chunks = [], the async acompletion example without stream=True, the __anext__() note enabling async iteration, the async streaming example, and the infinite-loop handling with litellm.REPEATED_STREAMING_CHUNK_LIMIT = 100, InternalServerError, the proxy's litellm_settings override and the reproduction snippet using ModelResponseListIterator and CustomStreamWrapper: docs.litellm.ai/docs/completion/stream
• [2] LiteLLM, Timeouts — timeout for the complete response with a 300 example, stream_timeout for the first chunk with a 30 example and its application to completion(), acompletion() and aresponses() on direct SDK calls, and keepalive_seconds sending SSE ping comments clamped to 1-300 seconds with 0 disabling it: docs.litellm.ai/docs/proxy/timeout
• [3] LiteLLM, Config settings — always_include_stream_usage defaulting to false and including usage metrics in every streaming response chunk, and sse_keepalive_ping_interval_seconds as the proxy-wide default for deployments and pass-through routes without their own keepalive: docs.litellm.ai/docs/proxy/config_settings

Related Articles

• LiteLLM Retries and Fallbacks: Stop Retrying a 401
• LiteLLM Errors: Your 401 Might Be a Budget Problem
• LiteLLM in Docker: The Salt Key You Can Never Change