LLM Gateways Compared: 9 Options, No Vendor Rankings

Contents

Figure 1: LLM Gateways Compared: 9 Options, No Vendor Rankings — one endpoint, nine very different sets of trade-offs

Your app calls three model providers, each with its own SDK, key, rate limits and outage schedule. Every new model means another integration, and every outage means a late-night patch. An LLM gateway promises to collapse all of that into one endpoint.

But which one? The market now includes open-source proxies, hosted marketplaces, edge services and API-management plugins. This guide compares the best LLM gateways and proxies, explains what each one does well, and warns you where the sales pitches get shaky.

A Quick Warning About "Best Gateway" Lists

Many top-gateway rankings come from companies that sell a gateway. One publisher put out several "top 5" lists that rank its own open-source gateway first, and other vendors do the same. Latency figures, like "11 microseconds of overhead", come from those vendors' own tests.

I used these pages for facts about features and pricing, and I checked pricing against vendor documentation where I could. I did not repeat any vendor's ranking. You should run your own test, and I show you how near the end.

A concrete example of why: Bifrost's own site quotes 11 microseconds of overhead in its FAQ and "20 Micro-s" in the hero section on the same page [2]. Both cannot be the number you should plan around.

What an LLM Gateway Actually Does

An LLM gateway sits between your apps and your model providers. It gives you one API, then adds the controls that raw provider APIs lack:

Control	What it gives you	Why it matters
Unified access	One request format for many providers	No per-provider SDK in every app
Reliability	Retries, fallbacks and load balancing	A provider outage becomes a routing event
Cost control	Budgets, rate limits and caching	One runaway job cannot burn the month
Visibility	Logs, token counts and spend by team or key	You can finally answer "who spent this?"
Governance	Access control, guardrails and retention rules	Prompts are data, and data has rules

Gateways split into three families, and picking the right family matters more than picking the right product.

Family 1: Self-Hosted Open-Source Gateways

You run these yourself. You get control over data and keys, and you accept the operational work.

LiteLLM

LiteLLM is the most widely used open-source gateway. It offers an OpenAI-compatible proxy for 100+ providers, with virtual keys, budgets, rate limits, caching, fallbacks and an Admin UI [1]. The core is MIT-licensed, and the official docs mark some features, like SSO, audit logs and per-model key budgets, as Enterprise [1].

The cost is operations. You run the proxy, a Postgres database and usually Redis. You also own security: two LiteLLM versions were published to PyPI with credential-harvesting malware in March 2026, so pin exact versions and follow the advisory [1].

• Best for: teams that want control, per-team budgets and a mature feature set
• Watch out for: operations burden and patch discipline

Bifrost

Bifrost is an open-source gateway written in Go by Maxim AI [2]. It is Apache-2.0, not MIT, and the repository dates from March 2025, which makes it the youngest project in this list [2]. The vendor claims very low overhead and combines LLM gateway, MCP gateway and agent gateway features [2].

Those performance numbers come from the vendor, so test them yourself. The site contradicts itself on the headline figure, which is a good reason to benchmark rather than quote.

• Best for: teams that care about raw throughput and want a newer, Go-based option
• Watch out for: a younger project with fewer years of production history than LiteLLM

Portkey's Open-Source Gateway

Portkey publishes an MIT-licensed TypeScript gateway that handles routing, retries, fallbacks, load balancing and guardrail checks [3]. Its hosted platform adds stored logs, analytics and governance, with a free Developer tier, a $49 per month Production tier and custom Enterprise pricing [3].

Ownership changed this year. Palo Alto Networks announced its intent to acquire Portkey on April 30, 2026 and closed the deal on May 29, 2026, folding it into the Prisma AIRS platform [3]. The open-source repository's last tagged release is v1.15.2 from January 12, 2026, and its last push was May 25, 2026, so verify activity before you build on the open-source version [3].

• Best for: teams that want observability and governance, and Palo Alto customers
• Watch out for: uncertainty about the open-source roadmap

Kong AI Gateway

Kong extends its API management platform with AI plugins, including token-based rate limiting, semantic caching, prompt compression and an advanced AI proxy [4]. The split is explicit on Kong's pricing page: some AI Gateway plugins are free, and a named list — AI Rate Limiting Advanced, AI Semantic Cache, AI Prompt Compressor, AI RAG Injector, AI Proxy Advanced, AI Sanitizer, AI Semantic Prompt Guard, AI AWS Guardrails and AI Azure Content Safety — sits in the paid tiers [4]. Konnect's AI Management plan starts at $25 per month plus usage [4].

One number worth knowing before a proof of concept: the free AI Gateway tier proxies 5 unique LLMs, measured hourly and enforced monthly [4].

• Best for: platform teams that already run Kong for regular APIs
• Watch out for: you inherit Kong's operational model, and the useful AI plugins need a paid tier

Family 2: Hosted and Managed Gateways

You sign up, get a key and run nothing. You trade control for convenience.

OpenRouter

OpenRouter is a hosted marketplace with one key for hundreds of models. Its pricing page lists 500+ models and 80+ providers on Standard [5]. It says it passes provider prices through with no markup on inference and charges 5.5% when you buy credits, or 8% on Business [5].

BYOK is free up to $25,000 of monthly list-price inference on pay-as-you-go, then 5% of what the same usage would have cost on OpenRouter [5]. I could not confirm the $0.80 minimum that older posts quote, so do not plan around it [5].

• Best for: fast access to many models, prototypes and small teams
• Watch out for: no self-hosting, a percentage fee at scale, and a governance ceiling that some comparison guides flag

Vercel AI Gateway

Vercel charges provider list prices with no markup and no platform fee on tokens [5]. The free tier includes $5 of monthly credit and a subset of models; the paid tier is pay-as-you-go with purchased credits [5].

Three details decide the real cost:

1. BYOK is available on the paid tier only, and it still requires purchased credits [5].
2. A failed BYOK request is retried with Vercel's system credentials, and that fallback usage is charged against your credit balance [5]. Your "free" BYOK path can therefore produce a bill.
3. Zero data retention is not the default across all plans: per-request ZDR costs nothing but is available on Pro and Enterprise, and team-wide ZDR costs $0.10 per 1,000 requests [5].

Budgets can be set per team, project, API key and team member [5].

• Best for: teams on Vercel and the AI SDK
• Watch out for: hosted-only, and BYOK requests that fall back to system credentials spend real credits

Cloudflare AI Gateway

Cloudflare's gateway runs on its edge network. Its core features — dashboard analytics, caching and rate limiting — are free on all plans [5]. Unified Billing adds a 5% fee to credits bought through Cloudflare, with no markup on provider token rates: a $100 purchase costs $105 [5].

The feature list is broad but young: dynamic routing, spend limits, guardrails, data loss prevention and BYOK are all marked beta in the docs [5]. Guardrails run on Workers AI with @cf/meta/llama-guard-3-8b, billed as token-based inference [5].

• Best for: teams already on Cloudflare who want near-zero setup
• Watch out for: beta features, lighter routing and governance than the heavier platforms, and no self-hosting

Family 3: Observability-First and Enterprise Platforms

Helicone

Helicone leads with logging and cost tracking, with a proxy built around that. Its pricing page lists a free Hobby tier with 10,000 requests and 1 GB of storage, a $79 Pro plan and a $799 Team plan, with Enterprise adding on-prem deployment and SAML SSO [5]. The repository is Apache-2.0 and was still being pushed to in October 2026 [5].

Many teams run it next to another gateway for failover, which is a reasonable pattern but means two hops to debug.

• Best for: teams that mainly want to see what their LLM traffic does
• Watch out for: it is an observability product first, so check that its routing and limits match your needs

TrueFoundry

TrueFoundry offers a managed platform with self-hosting in your own cloud or Kubernetes environment. Its pricing page lists a free Developer tier for up to 3 users with 10K requests and tool calls per user per month, a Pro tier, and an Enterprise tier with VPC, on-prem and data residency options plus an SLA [5].

• Best for: enterprises that want vendor support inside their own infrastructure
• Watch out for: custom pricing and a platform larger than a gateway alone

Side-by-Side Comparison

Gateway	Hosting	Open source	Cost model	Standout trait
LiteLLM	Self-host	Yes, MIT core	Free, plus infrastructure	Budgets, keys and breadth [1]
Bifrost	Self-host	Yes, Apache-2.0	Free OSS, paid enterprise	Vendor-claimed speed in Go [2]
Portkey	OSS gateway plus hosted platform	Gateway yes, MIT	Free OSS, hosted from $49/month	Observability and governance [3]
Kong AI Gateway	Self-host or Konnect	Core yes, AI plugins partly paid	Konnect Plus from $25/month plus usage	Fits existing Kong estates [4]
OpenRouter	Hosted only	No	5.5% Standard / 8% Business credit fee	Large model catalog [5]
Vercel AI Gateway	Hosted only	No	List price, no markup, $5 free credit	Fits the Vercel AI SDK [5]
Cloudflare AI Gateway	Hosted only	No	Core free, 5% on unified-billing credits	Edge caching and simplicity [5]
Helicone	Hosted, on-prem on Enterprise	Yes, Apache-2.0	Free 10k requests, $79 Pro	Observability first [5]
TrueFoundry	Your cloud, managed	No	Free Developer, custom Enterprise	Supported VPC deployment [5]

Provider and model counts differ wildly across vendor pages, and each one counts differently. I left them out of the table on purpose.

How to Choose: Start With Your Binding Constraint

Pick the question that matters most to you, then follow it:

1. Data must stay in your network: choose a self-hosted option like LiteLLM, Bifrost, Portkey's open-source gateway or Kong.
2. You have no one to run it: choose a hosted option like OpenRouter, Vercel or Cloudflare.
3. You already live on a platform: match it. Vercel on Vercel, Cloudflare on Cloudflare, Kong with Kong.
4. You need deep visibility: look at Portkey or Helicone, and compare them against LiteLLM's logging.
5. You need supported enterprise deployment: look at TrueFoundry, Portkey Enterprise or Kong Enterprise.
6. You need the widest catalog quickly: start with OpenRouter.

Most teams should begin with the simplest option that satisfies their data rules, then move up only when a real limit shows up.

If the shortlist is LiteLLM versus the hosted options, the LiteLLM vs OpenRouter vs Portkey comparison goes deeper on the fee maths.

Questions Worth Asking Every Vendor

Question	Why it decides the deal
What does it log, and for how long?	Prompts often contain sensitive data, and retention is a compliance decision
What happens when the gateway fails?	It becomes your single point of failure unless you plan for it
How does it count tokens and cost?	Budget accuracy depends on it, especially with caching and BYOK
Who maintains it?	Check release dates, ownership and funding
How do I leave?	Look for OpenAI-compatible APIs and keep your own model aliases

That last question is the cheapest insurance you can buy. A gateway you can swap in a weekend is a very different risk from one embedded in your config.

Run Your Own Bake-Off

Vendor benchmarks tell you little about your workload. Test two finalists with a plan like this:

1. Use the same providers and the same requests through both gateways.
2. Measure latency overhead against calling the provider directly, at your real request sizes.
3. Simulate a provider outage and watch the failover behaviour.
4. Check spend reports against actual provider invoices.
5. Test limits and budgets with real keys, not admin credentials.
6. Record the operational cost, meaning setup time and upgrade effort.

Then keep the loser's configuration in a branch. The bake-off is cheaper the second time.

Common Pitfalls

1. Trusting vendor rankings. The author of a list often sells the winner.
2. Choosing on features you will never use. Daily operations matter more.
3. Ignoring the gateway as a security target. It holds your provider keys, so it deserves the same review as your database.
4. Forgetting the exit plan. Lock-in sneaks in through proprietary configs.
5. Skipping the failure test. A gateway that works on sunny days proves little.
6. Assuming free tiers cover production. Kong's free AI Gateway proxies 5 unique LLMs, and Vercel's free tier is a model subset [4][5].
7. Assuming BYOK means no bill. Vercel retries failed BYOK calls on its own credentials and charges your credits [5].
8. Designing around beta features. Cloudflare's routing, guardrails, spend limits and DLP are all beta [5].

Wrapping Up

The best LLM gateway depends on your binding constraint. LiteLLM leads on open-source breadth and budgets, OpenRouter, Vercel and Cloudflare win on zero-ops convenience, Portkey and Helicone lean toward visibility, and Kong and TrueFoundry serve existing platform and enterprise needs.

Recent changes, like Portkey's acquisition and LiteLLM's security incident, show why you should check maintenance and ownership, not just the feature grid [1][3].

So write down your top three constraints, shortlist two gateways from the right family, and run the bake-off. A week of real traffic will settle what no ranking can.

If you pick LiteLLM, start with the proxy server guide, then cap spend with the token limits guide before you invite the second team in.

Frequently Asked Questions

Which LLM gateway is best overall?

There is no single best gateway, because the pick follows your binding constraint. Self-host LiteLLM if traffic and provider keys must stay in your network, use OpenRouter or Cloudflare if you have nobody to run infrastructure, use Kong if you already run Kong for APIs, and use Portkey or Helicone when traces and guardrails are the main requirement. Shortlist two from the right family and run a one-week bake-off with your own traffic.

Are LLM gateway rankings trustworthy?

Treat them as advertising. Several top-gateway lists are published by companies that sell a gateway and rank their own product first, and the latency numbers in those posts usually come from the vendor's own benchmark. I used vendor pages for features and prices, checked them against documentation where I could, and did not repeat any vendor's ranking. Test overhead on your own request sizes before you believe a microsecond figure.

Does OpenRouter charge a markup on tokens?

No. OpenRouter says it passes provider prices through with no markup on inference, and its fee is charged when you buy credits: 5.5% on the Standard plan and 8% on the Business plan, with discounts on Enterprise. BYOK is free up to $25,000 of monthly list-price inference on pay-as-you-go, then 5% of what the same usage would have cost on OpenRouter. I could not confirm the $0.80 minimum that older posts quote.

What does Vercel AI Gateway cost?

Vercel charges no markup and no platform fee on tokens: you pay the provider's list price. The free tier includes $5 of monthly credit and a subset of models, and the paid tier is pay-as-you-go with purchased credits. BYOK is available on the paid tier only and still requires purchased credits, and a failed BYOK request is retried with Vercel's own credentials and charged to your credit balance.

Is Cloudflare AI Gateway free?

Its core features are free on all plans, including dashboard analytics, caching and rate limiting. Unified Billing adds a 5% fee to credits bought through Cloudflare, with no markup on the provider's per-token rates. Guardrails, dynamic routing, spend limits, data loss prevention and BYOK are all beta features, so read the current docs before you design around them.

What happened to Portkey and LiteLLM in 2026?

Palo Alto Networks announced its intent to acquire Portkey on April 30, 2026 and closed the deal on May 29, 2026, making Portkey the AI gateway inside Prisma AIRS; the open-source gateway's last tagged release is v1.15.2 from January 12, 2026. LiteLLM had a supply-chain incident on March 24, 2026, when versions 1.82.7 and 1.82.8 were published to PyPI with credential-harvesting malware; PyPI quarantined them the same day, Docker images were unaffected, and a clean v1.83.0 followed.

Sources

• [1] LiteLLM, repository, docs and March 2026 security update — the LICENSE file marking everything outside enterprise/ as MIT, the docs banner listing SSO/SAML, audit logs, spend tracking and multi-team management as Enterprise, the per-model model_max_budget Enterprise marker, "call 100+ LLMs", and the incident report for litellm 1.82.7 and 1.82.8 with the clean v1.83.0 follow-up: github.com/BerriAI/litellm/blob/main/LICENSE, docs.litellm.ai and docs.litellm.ai/blog/security-update-march-2026
• [2] Bifrost by Maxim AI, product site and repository — the FAQ's "11 microseconds of overhead at 5k RPS on a t3.xlarge machine" claim against the hero section's "20 Micro-s", the combined LLM, MCP and agent gateway scope, the free and open-source positioning with paid enterprise features, and the repository's Go language, Apache-2.0 licence, March 2025 creation date and star count: getbifrost.ai and github.com/maximhq/bifrost
• [3] Portkey, pricing, repository and ownership — the MIT-licensed gateway, the free Developer tier with 10k recorded logs, the $49/month Production tier with 100k logs and +$9 per extra 100k, Enterprise VPC hosting, the April 30, 2026 acquisition announcement and May 29, 2026 completion, and the gateway repository's last tag v1.15.2 (January 12, 2026) with a last push on May 25, 2026: portkey.ai/pricing, paloaltonetworks.com, April 30, 2026, paloaltonetworks.com, May 29, 2026 and github.com/Portkey-AI/gateway
• [4] Kong, pricing and AI Gateway — the free versus paid AI Gateway plugin split with the named enterprise plugins (AI Rate Limiting Advanced, AI Semantic Cache, AI Prompt Compressor, AI RAG Injector, AI Proxy Advanced, AI Sanitizer, AI Semantic Prompt Guard, AI AWS Guardrails, AI Azure Content Safety), the AI Gateway Manager add-on, the 5-LLM proxy limit on the free tier measured hourly and enforced monthly, and Konnect's AI Management plan from $25/month plus usage: konghq.com/pricing and konghq.com/products/kong-ai-gateway
• [5] Hosted and platform vendors, pricing and docs — OpenRouter's no-markup inference, 5.5%/8% credit fees, $25,000 BYOK allowance and 5% above it, 500+ models and 80+ providers; Vercel AI Gateway's zero markup, $5 monthly free credit, model subset on free tier, paid-tier-only BYOK requiring purchased credits, BYOK failure retried with system credentials and charged to credits, per-request versus team-wide ZDR at $0.10 per 1,000 requests, and budgets per team, project, key and member; Cloudflare AI Gateway's free core features, 5% Unified Billing fee, no provider markup, beta dynamic routing, spend limits, DLP and BYOK, and guardrails on @cf/meta/llama-guard-3-8b; Helicone's 10,000-request free tier, $79 Pro, $799 Team and Enterprise on-prem with SAML SSO, plus its Apache-2.0 repository; and TrueFoundry's free Developer tier for 3 users with 10K requests per user per month and Enterprise VPC, on-prem and data residency options: openrouter.ai/pricing, vercel.com/docs/ai-gateway/pricing, developers.cloudflare.com/ai-gateway/reference/pricing, helicone.ai/pricing and truefoundry.com/pricing

Related Articles

• LiteLLM vs OpenRouter vs Portkey: Pick the Wrong One
• LLM Token Limits: The Settings That Actually Cut Spend
• LiteLLM Proxy Server: The Setup That Survives Week Two
• LiteLLM Load Balancing: Your RPM Limit Is a Suggestion