Semantic Caching for LLM Apps: Cut Latency and Spend

Contents

Meaning-keyed answers, the threshold that decides everything, and the four mistakes that bite first

Your chatbot just answered "how do I reset my password?" for the 400th time today. You paid for all 400 answers. Ouch.

I hit that wall on a support bot I built last year. Our bill climbed, our users waited three seconds per reply, and roughly 60% of the questions looked like cousins of each other. That's when I found semantic caching for LLM apps, and I haven't looked back.

This is the "semantic caching" lever the LLM cost guide lists as carefully — here's the full treatment: how it works, what it saves, and where it bites.

What Semantic Caching Actually Does

A normal cache stores an answer under an exact key. Same string in, same answer out. It's simple, fast, and nearly useless for human language.

A semantic cache stores answers under the meaning of the question. It turns each prompt into an embedding (a list of numbers that captures meaning), then searches for earlier prompts that sit close by. If it finds a close match, it returns the saved answer and skips the LLM call entirely.

Think of a barista who remembers your order even when you phrase it differently. "Large oat latte" and "big latte with oat milk" mean the same drink. No need to re-brew from scratch.

Figure 1: The barista remembers the order, not the exact sentence — that's the whole trick

Why Exact-Match Caching Fails With LLMs

Ever noticed how many ways people ask the same thing?

• "How do I reset my password?"
• "I forgot my password, help"
• "Password reset steps please"

An exact-match cache sees three strangers. It misses two of them and sends both to the model — you pay twice for one answer.

IMO, that makes exact-match caching a rounding error for chat-style apps. It works fine for fixed templates and batch jobs. It collapses the moment real humans start typing.

How Semantic Caching Works

The flow is shorter than you'd expect.

The Five-Step Loop

1. Embed the incoming prompt with an embedding model.
2. Search your vector store for the nearest stored prompts.
3. Compare the similarity score against your threshold.
4. On a hit, return the stored answer immediately.
5. On a miss, call the LLM, return the answer, and store the new prompt-answer pair.

Your cache gets smarter with every miss, and your LLM bill gets smaller with every hit.

A Minimal Code Sketch

The logic fits in a few lines:

def answer(prompt, system=None):
    vec = embed(prompt)                        # user's question only — see Mistake 2
    hit = cache.search(vec, threshold=0.92)    # start 0.90–0.95, tune by hand
    if hit:
        metrics.hit(prompt, hit.prompt, hit.score)   # log every hit for weekly review
        return hit.answer
    reply = llm(prompt, system=system)
    cache.add(vec, prompt, reply, ttl=ttl_for(topic(prompt)))
    metrics.miss(prompt)
    return reply

Choosing Your Building Blocks

You need three pieces. Pick each one deliberately:

• Embedding model: small and fast beats big and fancy here. The lookup must stay far cheaper than the LLM call, or you defeat the purpose.
• Vector store: Redis, pgvector, Qdrant, and FAISS all work. I like whatever you already run, because a new database means new headaches.
• Similarity metric: cosine similarity is the usual choice. Most tools default to it, and it behaves well.

The Payoff: Latency and Spend

Let's talk numbers. An LLM call often takes one to several seconds, depending on the model and output length. A cache lookup typically finishes in tens of milliseconds.

That gap changes how your app feels. Users get instant answers on repeated questions, and instant feels like magic. FYI: that speed boost matters more than most teams expect.

What Happens to Your Bill

Every hit skips the full token cost of generation. If 30% of your traffic hits the cache, you cut roughly 30% of your LLM spend (minus the tiny embedding cost). On my support bot, hit rates settled around 40% after a few weeks, and the savings paid for the whole project in a month.

Your mileage will vary. Hit rates depend on how repetitive your traffic looks. Ask yourself: do my users ask the same things over and over?

Where Semantic Caching Shines

App type	Verdict
Customer support and FAQ bots	Great fit
Internal knowledge-base assistants	Great fit
Documentation search and Q&A	Great fit
Educational tools with similar student questions	Great fit
Creative writing	Poor fit — you want fresh output
Highly personalized responses	Poor fit — depends on user data
Real-time data queries (stocks, scores)	Poor fit — stale by definition

The rule of thumb: if two users should get the same answer, cache it. If they shouldn't, skip it.

Tuning the Similarity Threshold

This part trips everyone up. The similarity threshold decides how close two prompts must sit before you treat them as the same question.

Set it too low, and your cache returns wrong answers. "How do I cancel my subscription?" might match "How do I upgrade my subscription?" — shared vocabulary, opposite wishes. Yikes.

Set it too high, and you barely hit the cache. You pay for embeddings and save nothing.

How I Pick a Starting Point

I start with cosine similarity around 0.90 to 0.95 and adjust from there, following this routine:

1. Log every cache hit with the original prompt, the matched prompt, and the score.
2. Review a sample of hits by hand each week.
3. Raise the threshold if you spot wrong matches.
4. Lower it slowly if you see too few hits and no errors.

Treat the threshold as a dial, not a constant. Your domain decides the right number, not a blog post — including this one. The same "measure, then tune" discipline the agent evaluation guide applies to judges applies here.

Keeping Your Cache Fresh

Stale answers hurt trust fast. Your product changes, your policies change, and your cache must keep up.

• Set a TTL on every entry. Short values for volatile topics, long for stable ones.
• Tag entries by topic or document version, so you can wipe a whole category when content updates.
• Namespace by model and prompt version. A new system prompt means new answers, so old entries should go — the same versioning rule the prompt versioning guide makes for prompts themselves.
• Add a manual purge button. You will need it at 4 p.m. on a Friday.

Also think about privacy. Never cache responses that contain personal data, and keep separate caches per tenant if you serve multiple customers. One leaked answer can ruin your week.

Mistakes I Made So You Don't Have To

I earned these lessons the hard way. Greatest hits list:

• Caching without context. My bot answered follow-ups like "what about the annual plan?" with the answer to a different earlier question. Short follow-ups depend on conversation history, so include relevant context in the embedding or skip caching for them.
• Embedding the whole template. I embedded the full prompt, including a long system message. Every vector looked nearly identical, and everything matched everything. Embed only the user's actual question.
• Forgetting cold starts. A fresh cache gives zero savings. Seed it with your top 50 questions and vetted answers on day one.
• Skipping quality checks. I trusted the hit rate and ignored accuracy. A high hit rate with wrong answers just annoys people faster.

How to Measure Whether It Works

Numbers keep you honest. Track these four metrics from day one:

• Cache hit rate: the share of requests your cache answers.
• Hit precision: the share of hits a reviewer marks as correct. Aim high, because wrong answers cost more than slow ones.
• p50 and p95 latency: compare hits against misses to see the real speedup.
• Cost per request: include embedding and vector store costs, not only LLM tokens.

Compare these against a control group if you can. Route 10% of traffic around the cache and check whether users notice any quality difference. If they don't, congrats: you just saved money for free.

Figure 2: Hit rate is the vanity metric — hit precision is the one that keeps your users trusting the bot

Tools Worth Trying

You don't need to build this from scratch. Several projects handle the heavy lifting:

• GPTCache: an open-source library built for exactly this job, with pluggable embeddings and stores.
• Redis semantic caching: great if Redis already runs in your stack — and the gateway layer in the LiteLLM tutorial covers the same caching role at the API boundary.
• LangChain and LlamaIndex integrations: handy for apps that already use those frameworks.
• DIY with pgvector: perfectly reasonable for small teams that want full control.

Start with the tool that fits your current stack. Switching later costs less than you think.

Your Quick-Start Checklist

1. Measure first. Log your prompts for a week and check how often similar ones repeat.
2. Start narrow. Cache one high-traffic feature, not the whole app.
3. Pick a conservative threshold, and review hits manually.
4. Track hit rate, latency, and cost on a dashboard.
5. Expand slowly as you build confidence.

Common Pitfalls

• Caching the system prompt into the embedding — everything matches everything, and precision dies.
• A threshold set once and never reviewed — it's a dial; your traffic drifts.
• Chasing hit rate over precision — wrong answers cost more than slow ones.
• One shared cache for every tenant — privacy incidents start here.
• Assuming follow-ups are standalone questions — embed context or skip them.
• Forgetting the cold start — seed the cache on day one or week one looks useless.
• Leaving stale entries after a policy change — TTLs, version tags, and a purge button exist for this.
• Never running the 10% control — it's the cheapest proof you'll ever get.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is semantic caching for LLM apps?

A semantic cache stores answers under the meaning of the question instead of an exact string. Each prompt is turned into an embedding, the vector store returns the nearest earlier prompts, and if one sits above your similarity threshold the cached answer is returned without calling the LLM at all. Misses call the model and store the new prompt-answer pair, so the cache grows with traffic. It works because humans ask the same question dozens of ways: an exact-match cache sees three strangers where a semantic cache sees one question, and every hit skips the full token cost of generation.

How do you choose a similarity threshold?

Start at cosine similarity 0.90–0.95 and treat it as a dial, not a constant: log every hit with the original prompt, the matched prompt, and the score; review a sample by hand each week; raise the threshold the moment you spot a wrong match; lower it slowly only if hits are scarce and precision is clean. Too low returns wrong answers — "cancel my subscription" matching "upgrade my subscription" shares vocabulary but wants opposite things. Too high and you pay for embeddings while saving nothing. Your domain decides the number, not a blog post.

When does semantic caching work poorly?

Three poor fits: creative writing (you want fresh output every time), highly personalized responses that depend on user data, and real-time queries like stock prices where any cached answer is stale by definition. The rule of thumb: if two users should get the same answer, cache it; if they shouldn't, skip it. Short follow-ups ("what about the annual plan?") depend on conversation history — include the relevant context in the embedding or skip caching for them — and never cache responses containing personal data, or keep separate caches per tenant.

How do you measure whether a semantic cache works?

Track four metrics from day one: cache hit rate (share of requests answered from cache), hit precision (share of hits a reviewer marks correct — wrong answers cost more than slow ones), p50 and p95 latency for hits versus misses, and cost per request including embedding and vector-store costs, not only LLM tokens. Then run a control: route 10% of traffic around the cache and check whether quality visibly changes. If users don't notice, you've cut the bill for free. On the support bot that started this, hit rates settled near 40% and the savings paid for the project in a month.

Wrapping This Up

Semantic caching stores answers by meaning, so similar questions share one LLM call. It cuts latency from seconds to milliseconds and trims your bill on every hit. The catch? You must tune the threshold, expire stale entries, and protect user privacy.

I think of it as the cheapest upgrade you can ship this month. So pull up your logs, count how many times your users ask the same question, and brace yourself. You'll probably wince at the number — and then you'll fix it.

Related Articles

• How to Reduce LLM API Costs: Token Budgets and Model Routing
• LiteLLM Tutorial: One API for Every LLM Provider
• What Is LLMOps? Complete Guide to Running LLM Apps in Production
• Evaluating AI Agents: Metrics, Benchmarks, and Test Strategies