Semantic cache

What is a semantic cache?

A semantic cache sits in front of a language model and keeps a record of the questions it has already answered. A new question is checked against that record. If a sufficiently similar question is in there, the stored answer comes back and the model is never called. If not, the model is called and the new question and answer are stored for next time.

The word semantic is what separates it from an ordinary cache. A normal cache only helps when two requests are byte-for-byte identical, and with free text that almost never happens. "How do I change my delivery address", "can I update where you ship to" and "wrong address on my order, help" are three different strings with one answer. A semantic cache compares meaning, so all three can land on the same stored reply.

Think of the colleague on the support desk who remembers that this question came in yesterday and copies the answer from that ticket instead of looking it up again.

How a semantic cache works

1. Embed the question. The incoming text becomes an embedding, a list of numbers that places similar meanings close together. This is a separate, cheap model call, not a call to the chat model.

2. Search the cached questions. A nearest-neighbour search over the stored embeddings returns the closest earlier question with a similarity score.

3. Compare against a threshold. If the score clears the threshold you set, the stored answer is returned as a hit. Portkey's gateway, for example, counts a cosine similarity above 0.95 as a hit by default; Redis's RedisVL library defaults to a cosine distance of 0.1, the same idea expressed as a distance.

4. Miss: call the model and store. The question goes to the model as usual, and the cache stores its embedding with the answer so the next similar question becomes a hit.

Most implementations match on the user's message and ignore the system prompt, because that part is identical for every request and would make everything look similar. The embedding call and the search add a little latency to every request, which each hit pays back many times over.

Where a semantic cache pays off

The pattern works where many people ask the same thing in different words and the correct answer is the same for all of them. Support chatbots and FAQ traffic are the obvious case: opening hours, the return policy, how to reset a password, what delivery costs. Internal assistants over stable documents are the other one: the HR handbook, the expense policy, the product catalogue, where the same colleagues ask the same questions every month and the documents rarely change.

It does not work for anything conversational. LiteLLM says so in its proxy documentation: semantic caching is designed for single-shot prompts, and on multi-turn or agentic traffic it will replay stale responses. Turn five of a conversation looks almost identical to turn four, so the cache keeps serving the answer to turn four.

A worked example

Say a webshop assistant handles 10,000 questions a month, each through a prompt with retrieved product and policy text, and a full model call costs around 2 cents including the answer. That is 200 euro a month without a cache.

Suppose four questions in ten find a match. Then 4,000 answers come from the cache and 6,000 still go to the model, so the model bill drops to about 120 euro. The cache has its own cost, an embedding call for each of the 10,000 questions, but embeddings are priced in cents per million tokens, so short questions put that in the small change. What you pay on top is hosting the store. Net saving: near 80 euro a month, or 40 percent of the bill.

Two things follow. The saving scales with volume and with the cost per call, so a cache on cheap, low-volume traffic is not worth the operational attention. And the second benefit is often worth more than the first: a hit returns in milliseconds instead of seconds, and the common questions keep getting answered when the model provider is slow or down.

Semantic cache versus prompt caching

The names invite confusion and the two solve different problems. Prompt caching is done by the model provider and matches on an identical prefix; Anthropic's documentation states that cache hits require 100 percent identical prompt segments. The model still runs and still writes a fresh answer, and what you save is the re-reading of the same long instructions, at about a tenth of the normal input price. A semantic cache is something you run yourself, it matches on a similar question rather than an identical prefix, and on a hit the model does not run at all.

They stack. Prompt caching makes every model call cheaper; a semantic cache reduces how many model calls happen. A support assistant with a long fixed policy prompt benefits from both.

What to watch out for with a semantic cache

Azure API Management puts the warning at the top of its policy reference: because semantic caching returns responses based on similarity, it can surface responses that are incorrect, outdated or unsafe for the current request. That happens in two ways.

False hits. "How do I cancel my subscription" and "how do I stop my subscription from being cancelled" share almost every word and score as very similar. So do "do you deliver to the Netherlands" and "do you deliver to France". The cache returns a confident, well-written answer to the wrong question, and nothing in the response flags it. Set the threshold conservatively and measure it on your own traffic rather than accepting a default. Azure recommends starting strict, around 0.05 on its scale, and warns that going above 0.2 leads to mismatches.

Stale hits. Your return window changes from 14 to 30 days, the policy document is updated, and the cache keeps serving last month's answer because the question did not change. Every cached answer needs a time-to-live, and the affected entries have to be cleared whenever the source documents change. Azure's store policy makes the duration mandatory for exactly this reason.

The controls that keep both in check:

• Never cache answers that are personalised or depend on who is asking. "Where is my order" has a different correct answer for every customer, and a hit here is a data leak.

• Scope the cache per user, customer or tenant when answers legitimately differ. Azure's vary-by element and RedisVL's tag filters exist for this.

• Log every hit with the original question, the matched question and the score, so you can audit false hits and tune the threshold from evidence.

• Set a short TTL on anything that follows a document, from minutes to a day, and invalidate when the document changes.

Where a semantic cache lives

You rarely build one from scratch. LLM gateways such as Portkey, LiteLLM and Azure API Management offer it as a setting, GPTCache from the team behind the Milvus vector database adds one inside your own code, and vector stores such as Redis or Qdrant ship one on top of their search.