Handling LLM API Rate Limits: Retries, Backoff, and Queues

Contents

Retry the right errors, back off with jitter, throttle early, and let queues take the spike

Your app worked perfectly in testing. Then launch day arrived, a few hundred users showed up at once, and your logs filled with 429 Too Many Requests. Fun times.

I hit that wall during a product demo last year. Our summarization feature fired 50 parallel calls the moment a customer uploaded a big document, and the provider shut the door on every one of them. The customer watched a spinner. I watched my career flash by.

This guide shows you how to handle LLM API rate limits with smart retries, exponential backoff, and queues. Your users will never notice the limits, and neither will your pager — reliability sibling of the LLMOps production guide.

Understand the Limits You're Fighting

Providers cap your usage to protect their infrastructure and to share capacity fairly. Most limits come in a few flavors, and you can hit any of them first:

• Requests per minute (RPM): how many calls you can send, regardless of size.
• Tokens per minute (TPM): how much text you can process, input and output combined.
• Concurrent requests: how many calls can run at the same moment.
• Daily or monthly quotas: hard ceilings tied to your plan or spend tier.

TPM trips teams up most often. Ten huge requests can exhaust your token budget while your request count looks tiny. Names and exact rules differ by provider, so read your provider's current docs.

Read the Response Headers

Most providers tell you how much room you have left. Look for headers that report remaining requests, remaining tokens, and reset times. Many also send a Retry-After header on a 429 response.

Use those signals. They beat guessing every time. FYI: logging them also gives you a free early-warning system — the same per-request telemetry the cost tracking setup records anyway.

Decide What to Retry

Not every error deserves a retry. Retrying the wrong failure wastes money and hides bugs.

Retry these:

• 429 (rate limited): the classic. Wait and try again.
• 5xx errors (server trouble): temporary outages often clear up quickly.
• Timeouts and dropped connections: network hiccups happen.
• Overloaded responses: some providers use a special code when capacity runs short.

Don't retry these:

• 400 (bad request): your request is broken, and sending it again won't fix it.
• 401 or 403 (auth problems): fix your credentials instead.
• Content policy rejections: the same input will fail the same way.
• Context-length errors: shorten the input first.

Exponential Backoff With Jitter

When a retry makes sense, you must decide how long to wait. Retrying instantly makes things worse, because you add load to an already stressed system.

Exponential backoff doubles the wait after each failure: 1 second, then 2, then 4, then 8. That gives the service time to recover. But there's a catch.

Why Jitter Matters

Picture 500 clients that all fail at the same moment. With plain backoff, they all retry together at 1 second, again at 2 seconds, and so on. You've built a retry stampede that hammers the API in synchronized waves.

Jitter fixes this by adding randomness. Each client waits a random time inside the backoff window, so retries spread out. IMO, jitter is the single most overlooked detail in retry code.

A Minimal Code Sketch

This example honors the server's Retry-After value when present and otherwise uses full jitter. RateLimitError stands in for whatever exception your client library raises:

import random
import time

def call_with_retry(fn, max_attempts=6, base=1.0, cap=30.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimitError as err:
            if attempt == max_attempts - 1:
                raise
            if err.retry_after:
                delay = err.retry_after + random.uniform(0, 1)
            else:
                delay = random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(delay)

Set Sensible Limits on Retries

Unlimited retries create their own disasters. Cap them in three ways:

• Maximum attempts: five or six is plenty for most apps.
• Maximum delay: cap the wait at 30 to 60 seconds.
• Total time budget: give up when the user's patience runs out, not when your loop finishes.

Throttle Before You Hit the Wall

Retries handle failure after it happens. Smart apps prevent most failures in the first place by throttling on the client side.

Use a Token Bucket

A token bucket refills at a steady rate and lets each request spend tokens. When the bucket runs dry, requests wait. Run one bucket for requests per minute and another for tokens per minute, and your app stays under the limit by design:

class TokenBucket:
    def __init__(self, per_minute, burst):
        self.rate = per_minute / 60.0
        self.cap = burst
        self.tokens = burst
        self.updated = time.monotonic()

    def acquire(self, cost=1):
        now = time.monotonic()
        self.tokens = min(self.cap, self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        while self.tokens < cost:            # dry bucket: wait here, not at 429
            time.sleep((cost - self.tokens) / self.rate)
            now = time.monotonic()
            self.tokens = min(self.cap, self.tokens + (now - self.updated) * self.rate)
            self.updated = now
        self.tokens -= cost

Estimate Tokens Before You Send

Count input tokens up front with your provider's tokenizer, and add your max_tokens setting as the worst-case output. This estimate lets your limiter reserve capacity honestly. Skip it, and one giant request can blow your whole minute's budget.

Cap Concurrency

A semaphore limits how many calls run at once. Set it below your provider's concurrency limit, with headroom for spikes. My 50-parallel-calls disaster would have vanished with a concurrency cap of eight.

Put Queues Between Your App and the API

Queues turn a traffic spike into a manageable line. Instead of firing requests directly, your app drops jobs on a queue, and workers pull them at a pace your limits allow.

Queues give you three big advantages:

• Smoothing: bursts flatten into steady throughput.
• Durability: jobs survive crashes and deploys when the queue persists them.
• Control: you decide who goes first, and how fast.

Separate Interactive From Background Work

Users waiting on a chat reply can't wait behind a nightly batch job. Use separate queues or priority levels for each. Reserve a slice of your rate limit for interactive traffic, and let background jobs use the rest.

Consider Batch APIs for Bulk Work

Many providers offer batch endpoints with higher limits and lower prices for non-urgent jobs. If your task tolerates a delay of hours, batching often beats a clever retry system. Check your provider's current options.

Figure 1: The queue absorbs what the limit would have rejected — spikes become a line you control

Build Fallbacks for Bad Days

Sometimes retries and queues aren't enough, because the provider itself struggles. Plan for that day before it arrives:

• Fallback model: switch to a smaller or alternate model when the primary one fails repeatedly.
• Fallback provider: route to a second vendor for critical flows, if your quality bar allows.
• Circuit breaker: after several straight failures, stop calling the API for a short cool-down. This protects both your app and the provider.
• Graceful degradation: show a friendly message, a cached answer, or a simpler feature instead of an error page.

Test your fallbacks for quality, not only availability. A fast wrong answer can hurt more than a slow right one — the same eval discipline the CI/CD guide runs before every deploy.

Use Fewer Tokens in the First Place

The best way to dodge rate limits is to need less capacity. Every token you save buys you headroom:

• Trim prompts and conversation history.
• Set tight max_tokens limits so replies don't run long.
• Cache repeated answers and prompt prefixes where it's safe — prefix caching is half the point of the semantic caching guide.
• Route simple tasks to smaller models, which often carry higher limits — the routing half of the cost reduction playbook.

Watch It Like a Hawk

You can't tune what you can't see. Add these signals to your dashboards:

• 429 rate per minute, split by feature and model.
• Retry counts and the share of requests that needed one.
• Queue depth and wait time.
• Remaining-capacity headers, logged from each response.
• Time spent waiting on limits, added to user-facing latency.

Alert on rising 429 rates and growing queue depth. Both warn you before users complain.

Mistakes I Made So You Don't Have To

Here's my greatest hits list:

• Retrying without backoff. My first retry loop hammered the API and prolonged the outage.
• Skipping jitter. All workers retried in lockstep, and the 429s kept coming.
• Retrying non-idempotent actions. A retry after a timeout repeated a tool call that sent an email. Twice. Use idempotency keys or check state first.
• Ignoring Retry-After. The server told me exactly how long to wait, and I guessed instead.
• Sharing one limit across everything. A background job starved our chat feature. Reserve capacity for interactive traffic.

Common Pitfalls

• Retrying everything — 400s, auth failures, and content rejections fail identically on the second attempt.
• No jitter — synchronized clients retry in waves and keep the provider's door shut.
• Infinite retry loops — cap attempts, delay, and total time budget, or the loop outlives the user.
• Throttling only requests, never tokens — TPM is the limit that surprises you on launch day.
• Concurrency above the provider's cap — the semaphore must sit below the limit, with headroom.
• One shared queue for chat and batch — background work starves interactive traffic.
• Fallbacks tested for uptime only — a fast wrong answer is worse than a slow right one.
• 429s invisible in dashboards — if retry share and queue depth aren't graphed, you're tuning blind.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What are LLM API rate limits?

Providers cap usage to protect their infrastructure and share capacity fairly, in four flavors: requests per minute (RPM — calls you can send regardless of size), tokens per minute (TPM — input plus output text), concurrent requests (calls running at the same moment), and daily or monthly quotas tied to your plan. TPM trips teams up most often: ten huge requests can exhaust your token budget while your request count looks tiny. Most providers also report headroom in response headers — remaining requests, remaining tokens, reset times, and often a Retry-After on 429 — so read them instead of guessing, and log them for a free early-warning system. Exact rules differ by provider, so check your provider's current docs.

Which errors should you retry from an LLM API?

Retry 429 (rate limited — wait, then try again), 5xx server errors (temporary outages often clear quickly), timeouts and dropped connections (network hiccups), and overloaded-capacity codes where the provider offers one. Never retry 400 bad requests (the request is broken), 401/403 auth failures (fix credentials instead), content-policy rejections (the same input fails the same way), or context-length errors (shorten the input first). When a retry makes sense, use exponential backoff — 1s, 2s, 4s, 8s — with jitter so synchronized clients don't retry in lockstep, honor Retry-After when sent, and cap everything: five or six attempts, a 30–60 second max delay, and a total time budget that gives up when the user's patience runs out.

How do you prevent rate limit errors before they happen?

Throttle on the client side so the wall is never hit. Run a token bucket for requests per minute and another for tokens per minute — the bucket refills steadily, and requests wait when it runs dry, keeping you under the limit by design. Estimate tokens before sending: count input with the provider's tokenizer and add your max_tokens setting as worst-case output, so the limiter reserves capacity honestly. Cap concurrency with a semaphore set below the provider's limit with headroom — a cap of eight would have erased a 50-parallel-calls disaster. Then put a queue between your app and the API: bursts smooth into steady throughput, jobs survive crashes, and you control priority — with interactive chat on its own queue ahead of nightly batch jobs, and bulk work on discounted batch endpoints where latency allows.

What should you do when the provider itself is down?

Build four fallback layers before the bad day arrives. A fallback model switches to a smaller or alternate model after repeated failures of the primary. A fallback provider routes critical flows to a second vendor when your quality bar allows. A circuit breaker stops calling the API for a short cool-down after several straight failures, protecting both your app and the provider. Graceful degradation shows a friendly message, a cached answer, or a simpler feature instead of an error page. Test fallbacks for quality, not only availability — a fast wrong answer can hurt more than a slow right one. And watch the signals: 429 rate per minute, retry share, queue depth, logged capacity headers, and time spent waiting on limits.

Wrapping This Up

Rate limits aren't a bug. They're a fact of life with LLM APIs, and good engineering makes them invisible. Retry the right errors, back off with jitter, throttle before you hit the wall, and queue the rest.

Start with the retry wrapper and a concurrency cap. Those two changes alone would have saved my demo. Your next launch day can feel boring, and boring is the best compliment an infrastructure engineer can get.

Related Articles

• LiteLLM Tutorial: Unified LLM API with Fallbacks and Routing
• Semantic Caching for LLM Apps: Cut Latency and Cost
• Reduce LLM API Costs: Token Budgets, Caching, and Model Routing
• What Is LLMOps? Complete Guide to Running LLM Apps in Production