Hosting an LLM Gateway: The Cluster You Don't Need

Contents

Figure 1: Hosting an LLM Gateway: The Cluster You Don't Need — the gateway is small, the state around it is not

You picked a self-hosted gateway, and now it needs a home. Run it on a spare VM, and one reboot takes every AI feature in your company offline. Run it on an oversized Kubernetes cluster, and you spend more time on infrastructure than on the gateway.

This guide covers the best hosting for self-hosted LLM gateways. I use LiteLLM's current production docs for the concrete numbers, because it is the most widely used open-source gateway. The patterns carry over to other self-hosted gateways, though sizing details will differ.

What a Gateway Needs From Its Host

A gateway looks like a small web service, but it carries unusual responsibilities. Every AI request in your company passes through it, and it holds your provider keys. That changes how you think about hosting.

Component	Why the gateway needs it
The gateway app	A stateless service you can run in several copies
PostgreSQL	Keys, budgets, teams and spend logs; the docs require it for authentication, key management and usage tracking [1]
Redis	Shared rate limits, router state and caching across instances [3]
TLS and private networking	So only the right clients reach it
A secrets manager	The master key, the salt key and provider keys [1]
High availability	Because a gateway outage becomes everyone's outage

Notice what the list leaves out: GPUs. A gateway forwards requests to model providers, so it mostly waits on the network. You size it for concurrency and connections, not for model inference.

How Big Does It Need to Be?

LiteLLM's production docs size the proxy per worker: 1 vCPU and 4 GiB of memory, as both requests and limits [1]. The figures multiply, and that multiplication is real rather than notional: a container running 8 workers wants 8 vCPU and 32 GiB [1].

The 4 GiB is a floor, not a target, and the reason is unusual [1]:

• Prisma runs its query engine as a separate process whose resident memory behaves as a high-water mark.
• It grows to fit the largest single statement the engine has ever executed, and glibc does not return that memory to the operating system.
• A pod's floor therefore ratchets up to its worst-ever write and stays there for the life of the worker.
• The largest statements come from spend logging with store_prompts_in_spend_logs enabled, because each row carries a full prompt and response.
• Provision below 4 GiB and one large write can push the pod past its limit and get it OOM-killed, which shows up as a crash loop under traffic that looks unremarkable on every other metric.

That is the detail most sizing guides miss. If you store prompts, treat 4 GiB as the minimum and give the pod headroom above it [1].

Database and Redis Sizing

The docs publish separate sizing tables by sustained request rate [2][3]:

Sustained RPS	PostgreSQL	Redis
Up to 1K	4 vCPU, 16 GB RAM, 200 GB SSD with 3000+ IOPS, 100–200 connections	2 vCPU, 8 GB
1K to 5K	8 vCPU, 32 GB RAM, 500 GB SSD with 5000+ IOPS, 200–500 connections	4 vCPU, 16 GB
5K+	16+ vCPU, 64 GB RAM, 1 TB+ SSD with 10000+ IOPS, 500+ connections plus a read replica	8+ vCPU, 32 GB+

Storage tracks retention, not throughput. Budget roughly 1 to 2 KB per logged request without prompts, and ten times that with store_prompts_in_spend_logs enabled [2].

Most teams run far below these rates, so treat them as ceilings for planning. The docs report a benchmark of 8 ms p95 gateway overhead at 1,170 requests per second across four instances, and 29 ms p95 with two instances [4]. That is the vendor's test setup, with a mock model behind an AWS load balancer, so measure your own [4].

Workers and Instances

Worker settings trip up many first deployments. The docs give two rules, and they are opposites [1]:

1. On Kubernetes, or anywhere a pod-level autoscaler reads CPU: run one Uvicorn worker per pod and scale horizontally. One process per pod keeps latency predictable, lets the autoscaler read the CPU threshold against a single process, and makes rolling restarts hitless [1].
2. On a single VM or a bare container with nothing scaling it: set the worker count to the machine's vCPU count, since nothing else will use those cores [1].

litellm --config config.yaml --port 4000 --num_workers 4

Database connections multiply across your setup. The docs give a worked example: 3 instances, each with 4 workers and 10 connections per worker, add up to 120 connections [1].

The number that catches teams is autoscaling. Once an autoscaler owns the replica count, the instance count in that formula is maxReplicas, not today's pod count. The litellm-helm chart defaults autoscaling.maxReplicas and keda.maxReplicas to 100, so a full scale-out asks for roughly 1,000 connections at the default pool limit of 10 — far past what a stock Postgres accepts [1].

Two more worker rules worth pinning to the wall [1]:

• Never combine --num_workers 1 with --max_requests_before_restart. With one worker the flag recycles nothing, and once the limit is hit the whole pod stops serving. Health probes count toward the limit too, so two probes every 15 seconds use up 11,520 requests a day.
• Background jobs register once per worker process, not once per pod. A pod with 4 workers runs four copies of every job, so a 10-replica deployment runs forty. Set LITELLM_JOB_ROLE=worker on one deployment so shared jobs run once, and note that without Redis there is nothing to elect through, so they run unguarded on every process.

Hosting Options Compared

Option	Best for	Main strength	Main weakness
Single VM with Docker Compose	Trials and small internal teams	Cheapest and simplest	One machine means one failure point
Managed container service	Small to mid-size production	Less operational work than Kubernetes	Platform limits and quirks
Kubernetes (managed)	Larger teams and strict uptime needs	Scaling, rolling updates, ecosystem	Real operational complexity
Platform-as-a-service	Quick pilots	Fast setup	Limited networking and data control
Serverless functions	Rarely a good fit	Pay per use	Poor match for long-lived connections and shared state

The last two rows reflect my own assessment, not a vendor claim. Test them against your needs.

Option 1: A Single VM With Docker Compose

This is the fastest path. Run the gateway, Postgres and Redis on one machine with Docker Compose, which the official quickstart supports. It works well for trials, personal use and small teams that can tolerate downtime.

Its weakness is obvious. A reboot, a full disk or a failed upgrade stops everything. If you stay here, keep backups of the database and the .env file, and know that the salt key has no rotation path: changing it makes stored provider credentials unreadable [1].

If you are deploying the container for the first time, the Docker production guide covers the image, keys and probes.

Option 2: Managed Container Services

Services like AWS ECS and Fargate, Google Cloud Run and Azure Container Apps run containers without cluster management. They fit gateways well because the app is a stateless container, and you point it at managed Postgres and Redis.

Check these details before you commit:

Check	Why it matters
Minimum instances	Avoid scale-to-zero; cold starts hurt an always-on gateway
Connection limits	Autoscaling adds instances, and every instance adds database connections [1]
Timeouts	Streaming responses run long, so confirm the platform's request time limits
Memory sizing	Follow the per-worker guidance, including the 4 GiB floor [1]

Option 3: Managed Kubernetes

EKS, GKE and AKS give you rolling updates, autoscaling and a mature ecosystem. The docs support Helm on all three, and the install is the same on every cloud — only the data stores and ingress differ [5].

Outside Kubernetes, AWS and GCP have official Terraform modules that stand up the whole stack. Azure has no Terraform module, so AKS with Helm is the supported path there [5].

resources:
  requests:
    cpu: "1"
    memory: "4Gi"
  limits:
    cpu: "1"
    memory: "4Gi"

That snippet follows the docs' per-worker guidance for a single-worker pod [1]. Kubernetes shines when you already run a cluster and have people who know it. Spinning up a cluster only for a gateway usually costs more than it saves.

Scale on CPU and leave the memory target unset. The docs explain why: the query engine's resident memory reflects the largest write a pod has ever done, so a memory target ratchets replicas up after one large write and never scales them back in [1]. Their suggested CPU target is 60 rather than a higher threshold, because a replica is not useful the moment it is created [1].

Option 4: Platform-as-a-Service

Platforms like Railway, Render and Fly.io offer push-to-deploy hosting. They suit a quick pilot or a side project. I did not verify their current gateway-specific templates, so check each platform's documentation. For production, confirm private networking, database options and compliance support first.

Option 5: Serverless Functions

A gateway holds connections open for streaming, shares counters across instances and needs steady database connections. Function platforms fit that pattern poorly. If you want serverless economics, a managed container service with a minimum instance count comes closer.

Use Managed Postgres and Redis

Here is the best advice in this guide: run the gateway yourself, but let your cloud run the stateful pieces. A managed database gives you backups, failover and patching for free. Managed Redis does the same for your shared counters.

The docs recommend ElastiCache with the Valkey or Redis OSS engine at 7.x or newer on Graviton nodes, with cache.m7g.large up to 1K RPS and cache.m7g.xlarge above it, and they list Memorystore for GCP [3]. Run Redis 7.0 or newer in every case [3].

Two configuration notes from the docs:

1. Wire Redis in two places, not one. router_settings covers router state (cooldowns, usage and latency routing) and cache_params covers response caching plus the proxy-wide state — rate limits, budgets, cache invalidation and the pod lock. Setting REDIS_HOST and REDIS_PORT in the environment is not enough on its own; the proxy reads them only when the config points at Redis [3].
2. Set DISABLE_SCHEMA_UPDATE=true on application pods when migrations run separately, so pods do not race to migrate; the Helm PreSync hook runs them with DISABLE_SCHEMA_UPDATE=false [1].

router_settings:
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD

A correction to older advice you will find in blog posts: the current sizing page says to size Redis so eviction does not happen during normal operation, and to prefer adding capacity over relying on an eviction policy, because rate limit counters and queued spend updates are live accounting state [3]. An instance that evicts under pressure is dropping spend updates and resetting limit windows, not just losing cache hits. If you enable the Redis transaction buffer, enable persistence too, so a failover does not discard queued spend updates [3].

Make It Reliable

Hosting choice matters less than a few reliability habits.

Run More Than One Copy

Two instances behind a load balancer survive a crash and let you upgrade without downtime. The benchmark shows the difference: two instances produced 29 ms p95 gateway overhead, four instances produced 8 ms [4]. The docs also found that workers equal to CPU count gave the best performance in their single-VM tests [1].

Remember that multiple instances need Redis, or your rate limits count per instance [3].

Plan for Hitless Restarts

The docs describe recycling workers after a set number of requests with --max_requests_before_restart, and they discuss keeping restarts hitless [1]. On Kubernetes, use health checks and rolling updates so one pod drains at a time — and keep the single-worker rule from above in mind.

Think Before You Fail Open

The docs include an allow_requests_on_db_unavailable setting that keeps serving traffic when the database goes down, and they say to use it only inside a VPC that cannot be reached from the public internet [1]. When it is on, requests continue with a restricted internal-user fallback identity instead of failing, which means budgets and key checks are not enforced [1].

That trades safety for availability. Decide it deliberately. For hard limits, the docs also describe fail-closed settings for budgets and rate limits that reject traffic with a 503 when verification fails [1].

Lock It Down

Your gateway holds every provider key. Treat its host like a production database.

• Keep it private. Use private subnets, a VPN or a zero-trust proxy, and expose only what clients need.
• Terminate TLS at a load balancer or reverse proxy.
• Use a secrets manager for the master key, salt key and provider keys, not plain environment files in a repo.
• Set trusted_proxy_ranges to your load balancer or ingress CIDRs, or to [] when clients connect directly. Without it, the proxy warns at startup and only the weaker per-username sign-in limit applies [1].
• Restrict outbound traffic to your model providers where possible.
• Pin image versions, and consider pinning by digest. In March 2026, attackers published malicious LiteLLM versions on PyPI, and an unpinned install could have pulled them. Read the advisory, and rotate keys if you ran the affected versions.
• Limit Admin UI exposure. The master key is the Admin UI login password, so the console deserves the same care as a production console [1].

Watch It

A gateway you cannot see fails quietly. Monitor these basics:

1. Latency and error rates at the load balancer.
2. Database connections, since they scale with instances and workers [1].
3. Redis health and memory [3].
4. Spend log growth, because logs pile up in Postgres. Budget 1 to 2 KB per request without prompts and ten times that with prompts stored [2].
5. Cache hit rate, if you enabled caching.

The docs also suggest setting LITELLM_LOG=ERROR, set_verbose: False and json_logs: true in production [1]. And batch spend writes with proxy_batch_write_at: 60, because per-request writes become a database hot spot; above roughly 1,000 RPS, route those writes through Redis with the transaction buffer [1].

Keep Hosting Costs Reasonable

Hosting costs usually stay small next to model spend, but waste still adds up:

• Right-size from real usage. Start near the recommended minimum, then adjust.
• Avoid overbuying the database. Match the sizing table to your actual request rate [2].
• Share a cluster instead of building a new one for the gateway.
• Compare against a hosted gateway. If you spend a few engineer-days a month on upkeep, a hosted option may cost less overall — the nine-gateway roundup has the pricing.

Pick by Situation

1. Trying it out: Docker Compose on one VM.
2. Small production, no platform team: a managed container service with managed Postgres and Redis.
3. You already run Kubernetes: deploy there with the official Helm guidance.
4. Strict data residency or on-prem rules: run in your own VPC or data centre with the same components.
5. Nobody wants to run infrastructure: reconsider self-hosting and look at a hosted gateway.

Common Pitfalls

1. Running one instance in production. Every deploy then becomes an outage.
2. Skipping Redis with multiple instances. Your limits multiply by your instance count [3].
3. Ignoring database connection math. More pods mean more connections, and Postgres runs out.
4. Sizing a multi-worker container from the single-worker numbers. Eight workers need 8 vCPU and 32 GiB, not 1 and 4 [1].
5. Using latest image tags. You lose control over what runs.
6. Exposing the Admin UI to the internet. It is the master-key login, so treat it like a production console [1].
7. Forgetting backups for the .env and salt key. You cannot recover stored provider keys without them [1].
8. Leaving maxReplicas at the chart default. A hundred replicas at the default pool limit asks for 1,000 connections [1].
9. Relying on an eviction policy for Redis. Eviction drops spend updates, not just cache entries [3].

Wrapping Up

The best hosting for a self-hosted LLM gateway is the simplest setup that still gives you more than one instance, managed Postgres and Redis, private networking and pinned versions. Start on one VM to learn, move to a managed container service or Kubernetes for production, and let your cloud handle the stateful parts.

The numbers that matter are small and specific: 1 vCPU and 4 GiB per worker, connections equal to pool size times workers times replicas, and Redis before your second instance.

So sketch your target setup on one page: instances, database, Redis, network and secrets. Then size it from the docs' numbers, test a failure, and fix what breaks. A gateway that survives a deliberate crash will survive a real one.

If you are still choosing the gateway itself, the LiteLLM vs OpenRouter vs Portkey comparison covers the build-versus-buy maths, and the HTTP timeouts guide covers the tuning that comes after the host is settled.

Frequently Asked Questions

How much CPU and RAM does a LiteLLM proxy need?

The production docs size it per worker: 1 vCPU and 4 GiB of memory, as both requests and limits, multiplied by the worker count. A container running 8 workers wants 8 vCPU and 32 GiB. The 4 GiB is a floor rather than a target, because Prisma's query engine holds its resident memory at the high-water mark of the largest statement it has ever run, so a pod can ratchet upward after one big spend-log write and never release it.

Do I need Redis for a self-hosted LLM gateway?

Yes as soon as you run more than one worker or replica. Redis shares rate limit counters, router state, budgets and the response cache. Without it, each worker keeps its own copy, so a 100 requests-per-minute limit across four workers really allows 400, revoked keys stay usable on workers that missed the revocation, and cache hits only land on the worker that stored them.

What size Postgres and Redis does LiteLLM need?

The docs size by sustained request rate. Up to 1K RPS: Postgres with 4 vCPU, 16 GB RAM, 200 GB SSD and 100-200 connections, plus Redis with 2 vCPU and 8 GB. From 1K to 5K RPS: 8 vCPU, 32 GB and 200-500 connections, plus Redis with 4 vCPU and 16 GB. Above 5K RPS: 16+ vCPU, 64 GB, 1 TB+ and a read replica, plus Redis with 8+ vCPU and 32 GB+.

Should I run one worker or many in a LiteLLM container?

On Kubernetes, run one Uvicorn worker per pod and scale pods horizontally, so the autoscaler reads one process and rolling restarts stay hitless. On a single VM or a bare container with nothing scaling it, set the worker count to the machine's vCPU count. Never combine one worker with --max_requests_before_restart: with a single worker the flag does not recycle anything, and the pod stops serving when the limit is hit.

Is allow_requests_on_db_unavailable safe?

Only inside a VPC that cannot be reached from the public internet, which is what the docs require. When it is on and Prisma cannot reach the database, requests continue with a restricted internal-user fallback identity instead of failing, so budgets and key checks are not enforced. It trades safety for availability, and you should decide that trade deliberately rather than enable it by default.

Can I run a LiteLLM gateway on serverless functions?

It fits badly. The gateway holds streaming connections open, shares counters across instances through Redis, and keeps steady database connections, while function platforms are built for short-lived invocations. If you want pay-per-use economics, a managed container service with a minimum instance count above zero is the closer match.

Sources

• [1] LiteLLM, Production best practices — per-worker sizing at 1 vCPU and 4 GiB with the real multiplication for 8 workers, the 4 GiB floor and Prisma high-water-mark explanation with the store_prompts_in_spend_logs link, one-worker-per-pod on Kubernetes versus vCPU-count workers on a single VM, the --num_workers 1 plus --max_requests_before_restart warning with the 11,520 daily probe requests, the 3 × 4 × 10 = 120 connection example and the maxReplicas: 100 → ~1,000 connection trap, LITELLM_JOB_ROLE, CPU-only autoscaling with the 60 percent target, proxy_batch_write_at, database_connection_pool_limit, trusted_proxy_ranges, LITELLM_LOG=ERROR with set_verbose and json_logs, the salt key's one-way nature, DISABLE_SCHEMA_UPDATE=true on pods with the Helm PreSync hook, and allow_requests_on_db_unavailable limited to a VPC with the internal-user fallback: docs.litellm.ai/docs/proxy/prod
• [2] LiteLLM, Database sizing — the request-rate table (4 vCPU/16 GB up to 1K RPS, 8 vCPU/32 GB from 1K to 5K, 16+ vCPU/64 GB above with a read replica), connection counts per tier, storage at 1-2 KB per logged request and 10x with prompts, the note that connections break before CPU does, the database_connection_pool_limit × workers × instances formula, and cloud max_connections defaults for RDS, Aurora, Azure flexible server and Cloud SQL: docs.litellm.ai/docs/proxy/db_sizing
• [3] LiteLLM, Redis sizing, Redis requirements and Redis caching — the Redis table (2 vCPU/8 GB up to 1K RPS, 4 vCPU/16 GB to 5K, 8+/32 GB+ above), Redis 7.0 or newer, the advice to size so eviction does not happen and to prefer capacity over an eviction policy because counters and queued spend updates are live accounting state, persistence when the transaction buffer is on, ElastiCache with Valkey or Redis OSS on Graviton with cache.m7g.large and cache.m7g.xlarge and Memorystore on GCP, the per-worker multiplication without Redis and what degrades (rate limits, budgets, cooldowns, routing, cache hits, revoked keys), the need to configure both router_settings and cache_params rather than only environment variables, max_connections in cache_params, and REDIS_URL versus host/port/password: docs.litellm.ai/docs/proxy/redis_sizing, docs.litellm.ai/docs/proxy/redis_requirements and docs.litellm.ai/docs/proxy/caching_redis
• [4] LiteLLM, Benchmarks — the short-prompt chat result of 1,170 requests per second with 8 ms p95 gateway overhead across four instances versus 29 ms p95 with two, the 4 vCPU and 8 GB per-instance test deployment, the 3,000 RPS high-throughput run at 54 ms p95 with 33 pods of 4 workers and 16 GiB, and the methodology notes including the in-process mock model and public load balancer: docs.litellm.ai/docs/benchmarks
• [5] LiteLLM, Deploy — the two supported paths, Helm on EKS, GKE or AKS with the same install on every cloud, official Terraform modules for AWS and GCP, and Azure having no Terraform module so AKS with Helm is the supported path: docs.litellm.ai/docs/proxy/deploy

Related Articles

• LiteLLM in Docker: The Salt Key You Can Never Change
• LiteLLM Timeouts: The Setting Your Load Balancer Kills
• LiteLLM vs OpenRouter vs Portkey: Pick the Wrong One
• LLM Gateways Compared: 9 Options, No Vendor Rankings