How to Reduce LLM API Costs: Token Budgets and Model Routing

Contents

Six levers, the math behind each, and the one that bites if the checker is weak

Your LLM bill doesn't climb because of one big mistake. It climbs because every request carries a little extra, and requests add up. So where does the money actually go, and what can you cut without wrecking quality?

A small simulation checked the mechanics, then the providers' docs were read for the real discount rules. One warning up front: the prices in the simulation are made-up round numbers, not any provider's real rates. They show the shape of the savings, not your invoice. The ops context for all of this lives in the LLMOps production guide, where cost control is its own component.

Know What You Pay For

Every call costs roughly this:

cost = input tokens × input price + output tokens × output price

Output tokens usually cost more per token than input tokens. In the simulation, output was set at five times the input price — a placeholder, so check your provider's actual ratio. If you can't say how many tokens you send per request, start there. Log it before you optimize anything.

import logging
log = logging.getLogger("llm-cost")

PIN, POUT = 3.0 / 1e6, 15.0 / 1e6   # $ per token — placeholder, load from config

def chat(client, model, messages, **kw):
    r = client.chat.completions.create(model=model, messages=messages, **kw)
    u = r.usage
    log.info("llm_call", extra={"model": model, "in": u.prompt_tokens,
                                "out": u.completion_tokens,
                                "cost": u.prompt_tokens * PIN + u.completion_tokens * POUT})
    return r   # every dashboard in this article starts with this one log line

Lever 1: Stop Resending the Whole Chat

Chat apps resend the full history every turn. That's the sneaky one, because the cost grows with the square of the conversation length.

In the simulation, a 40-turn chat had a 1,500-token system prompt, 150-token user messages and 250-token replies. Resending everything meant 378,000 input tokens for just 10,000 output tokens — the closed-form formula matched the count exactly. Trimming history to the last few exchanges helped a lot:

Strategy	Input tokens	Cost vs full history
Full history	378,000	1.00
Last 10 exchanges	204,000	0.59
Last 4 exchanges	126,000	0.41

The catch: the table can't show what the model forgets. Trim blindly and your assistant asks users to repeat themselves. A middle path is to keep the last few turns verbatim and replace older turns with a short summary — summarization quality wasn't tested here, so try it on your own conversations.

Figure 1: Every turn resends the whole stack — the bill grows with the square of the chat

Lever 2: Cache the Part That Never Changes

If every request starts with the same system prompt, tool definitions or reference document, you're paying to reprocess it each time. Prompt caching discounts that repeated prefix.

What the docs say:

• OpenAI: caching applies to the unchanged beginning of a prompt — "cache reuse requires the entire rendered prefix to match." The minimum cacheable length is 1,024 tokens on current models. Cache reads cost 0.1× the uncached rate on most listed models, and writes cost 1.25×.
• Anthropic: you add a cache_control field. The default cache lifetime is 5 minutes, refreshed each time the content is used, with a 1-hour option. Writes cost 1.25× the base input price, reads 0.1× (lower on a few models). The usage fields cache_creation_input_tokens and cache_read_input_tokens show what happened.

Both providers also say caching has minimum lengths that vary by model, and prompts shorter than that simply aren't cached, with no error.

Does it pay off? Break-even math, using a 1.25× write and 0.1× read: a prefix used twice within its lifetime already beats not caching (0.675 of the cost). Used once, it costs 25% extra. At 10 uses it falls to 0.215, and at 100 uses to 0.112.

In the chat simulation, caching the growing history cut cost to 0.22 of the uncached cost, with 98% of input tokens served from cache. That's an upper bound — write premiums and cache expiry were ignored, and real traffic has gaps.

Practical rules from the OpenAI page:

messages = [
    {"role": "system", "content": STABLE_PROMPT},      # unchanged prefix: cacheable
    *history,                                          # append; never rewrite earlier turns
    {"role": "user", "content": latest_question},      # dynamic content last
]

• Put stable instructions first and dynamic content (timestamps, user text) last.
• Append new messages instead of rewriting earlier ones. Truncation or summarization can break reuse.
• Keep tool definitions stable.

Notice the tension with Lever 1: trimming history breaks the cached prefix. Pick one strategy per use case, or measure both.

Lever 3: Budget Your Output Tokens

Output is the expensive side, so cap it. A max_tokens limit is a blunt tool, and the simulation shows the trade. With reply lengths averaging a few hundred tokens (a lognormal distribution):

Cap	Replies cut off	Total cost vs no cap
200	69.3%	0.71
400	36.0%	0.84
600	19.1%	0.90
1,000	6.5%	0.96

A cap that cuts 36% of replies mid-sentence for a 16% saving is a bad deal for most products. FYI: the gentler alternative — asking for brevity in the prompt — wasn't measured, so test it yourself. Use hard caps as a safety net against runaway outputs, not as your main budget.

Lever 4: Batch Anything That Can Wait

If a job doesn't need an instant answer, use a batch API. OpenAI's docs list a "50% cost discount compared to synchronous APIs," a 24-hour completion window ("and often more quickly"), and separate rate limits. A single batch can include up to 50,000 requests.

Good fits: nightly classification, bulk summarization, evaluation runs, data labeling. Bad fits: anything a user is waiting on.

Lever 5: Route Easy Queries to Cheaper Models

The big one, and the riskiest. Not every request needs your most capable model. A cascade tries a cheap model first and escalates only when the answer looks unreliable.

The FrugalGPT paper describes three strategies: prompt adaptation (shorter prompts), LLM approximation (cheaper stand-ins like caches or smaller fine-tuned models) and LLM cascade. Its abstract claims it can match the best single model's performance "with up to 98% cost reduction," or improve accuracy by 4% at the same cost — the authors' results on their benchmarks. Don't expect them on your workload without testing.

The simulation uses a constructed world: query difficulty is uniform, the small model is right with probability 1 − 0.8 × difficulty, and the large model is right 95% of the time. The small model costs 1/20 as much:

Strategy	Quality	Cost vs always-large
Always large	0.950	1.00
Always small	0.600	0.05
Cascade, perfect checker	0.980	0.45
Cascade, 90% checker	0.939	0.47
Cascade, 70% checker	0.856	0.51
Cascade, 50% checker	0.774	0.55

The perfect-checker row matches a hand calculation (40% escalated). The 50% row is a coin flip, so quality lands at the average of the two models.

The lesson is blunt: the cascade is only as good as the checker. A 90% checker costs about 1 point of quality versus always-large. A 70% checker costs about 9. And the perfect checker beating the large model is an artifact of the setup, not something to expect in real life.

Also, a checker that is itself an LLM call adds cost. When it cost 1× a small call, the cascade rose from 0.45 to 0.50 of large-only cost. At 2×, it hit 0.55.

How to build a checker that's actually useful:

• Use deterministic checks first: valid JSON, schema match, required fields.
• Use confidence signals your task supports.
• Measure it against labeled examples before trusting it.
• Track escalation rate in production. A rising rate means your cheap model is slipping — the same drift signal the agent evaluation guide treats as a first-class metric.

Figure 2: A cascade is a toll road with an exit ramp — the checker decides who pays which toll

Lever 6: Shrink the Prompt

FrugalGPT calls this prompt adaptation, such as keeping only a few in-context examples. Practical moves:

• Cut redundant instructions.
• Use fewer few-shot examples, and prove they still help.
• Retrieve only the relevant chunks, not whole documents.
• Drop tool definitions a request can't use — careful, since the OpenAI docs advise keeping tool definitions stable for caching.

Again, the levers interact. Measure after each change. For the tool-calling half of this, the LiteLLM tutorial covers the same routing and cost-tracking machinery in code.

A Simple Order of Attack

1. Measure tokens per request, cost per user action and cache hit rate.
2. Cache the stable prefix.
3. Batch whatever can wait.
4. Trim history and prompts, then re-run your evals.
5. Cap output as a safety net.
6. Route last, because it needs a trustworthy checker and eval set.

IMO, routing gets the attention, but caching and batching are the safest wins — they don't change the model's answer.

The feature that taught me this was a document-chat widget that quietly re-embedded a 40-page PDF into every single turn. Median input was 21,400 tokens per request and the bill hit $1,840 in the first month. The logs told the story in an afternoon: 94% of those input tokens were byte-identical across calls. Caching plus a 6-turn history window brought the same traffic to $410, with no change to how the answers read. The bill wasn't a modeling problem. It was the same prefix, paid for over and over.

Common Mistakes

• Optimizing before logging. You can't cut what you can't count — start with tokens per request.
• Trimming history and enabling caching at the same time, then wondering why the cache hit rate is near zero. The levers fight each other.
• Copying someone's savings numbers. The simulations here are constructed worlds, and FrugalGPT's 98% is the authors' benchmark, not a promise.
• Routing without validating the checker. A weak checker silently trades quality for dollars.
• Using output caps as the main budget, cutting good replies to save a rounding error.
• Batching a user-facing job, then discovering the 24-hour window the hard way.
• Skipping the eval re-run after each change. Every lever has a price in quality; pay it deliberately.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Where does LLM API spending actually go?

Into input tokens and output tokens: cost = input tokens × input price + output tokens × output price. Output tokens usually cost more per token than input ones — in the simulation here, five times more, which is a placeholder, so check your provider's real ratio. The first move isn't optimization, it's visibility: log the token count of every request before you change anything. Most teams discover their bill is dominated by repeated prefixes (the same system prompt re-sent every turn) and by long outputs, not by the number of calls.

Does prompt caching actually pay off?

Usually, if the prefix is long and reused. Caching discounts a repeated prompt prefix: on OpenAI, cache reads cost 0.1× the uncached rate and writes 1.25×, with a 1,024-token minimum on current models and a strict rule that the entire rendered prefix must match. Anthropic uses an explicit cache_control field with a 5-minute lifetime (refreshed on use) or a 1-hour option, same 1.25× write and 0.1× read. Break-even math: a prefix used twice within its lifetime already beats not caching (0.675 of the cost); used once it costs 25% extra; at 10 uses it's 0.215, at 100 uses 0.112. Keep the stable prefix stable — timestamps and user text go last, and truncating history breaks reuse.

When does model routing save money without ruining quality?

When your checker is good. A cascade tries a cheap model first and escalates when the answer looks unreliable. In the simulation (small model at 1/20 the cost), a cascade with a 90% checker cost 0.47 of always-large for 0.939 quality versus 0.950 — about a point of quality for half the bill. With a 70% checker it fell to 0.856 quality for 0.51 cost: nine points lost. The checker itself costs money — an LLM-based checker added 1× a small call raised the cascade from 0.45 to 0.50. Build the checker from deterministic checks first (valid JSON, schema, required fields), validate it against labeled examples, and watch the escalation rate in production: a rising rate means your cheap model is slipping. Route last, after caching and batching.

What is the safest order to cut LLM costs?

Measure first (tokens per request, cost per user action, cache hit rate), then take the levers in order of risk: cache the stable prefix, batch whatever can wait, trim history and prompts and re-run your evals, cap output as a safety net against runaway replies, and route only at the end because it needs a trustworthy checker and eval set. Caching and batching are the safest wins because they don't change the model's answer. Routing gets the attention, but it's the one lever most likely to hurt quality if the checker is weak. Re-run your evals after every change — each lever trades quality or complexity for dollars.

Wrapping This Up

Costs come from tokens in and tokens out. Cut repeated input with caching, stop resending needless history, batch the patient work, cap runaway output and route easy queries to cheaper models. Each lever has a price in quality or complexity, and the simulation shows the routing lever is the one most likely to hurt if the checker is weak.

Start tonight: log tokens per request and see where the bill really comes from. Then pick the one lever that fits, and re-run your evals after it.

Related Articles

• LiteLLM Tutorial: One API for Every LLM Provider
• What Is LLMOps? Complete Guide to Running LLM Apps in Production
• LLMOps vs MLOps: What Changes When You Ship LLM Applications
• Evaluating AI Agents: Metrics, Benchmarks, and Test Strategies