Prompt caching

What is prompt caching?

Prompt caching lets a model provider store the processed form of the front part of your prompt and reuse it on the next call. The model still reads the whole prompt. What disappears is the repeated work of turning the same thousands of tokens of instructions and reference material into internal state, over and over, on every single request.

The saving lands on the invoice. AI APIs bill per token, and the stable front of a prompt is usually the biggest part of it: the role description, formatting rules, worked examples, a product catalogue, a policy document. A support assistant might send 15,000 identical tokens ahead of a 40-token question. Without caching you pay for 15,040 tokens on every call. With caching you pay a steeply discounted rate for the 15,000 and full price for the 40.

It helps to think of it as a bookmark rather than a memory. The provider does not remember your conversation. It remembers how far it got while processing one specific sequence of tokens, and it can pick up from there only when the next request opens with exactly that sequence.

How the prefix cache works

Caching matches on prefixes, not on meaning. The provider hashes the opening tokens of your request and looks for a stored entry with the same hash. Azure OpenAI documents the mechanics plainly: requests are routed on a hash of the initial prefix, typically the first 256 tokens; a request needs at least 1,024 tokens to be eligible; and the first 1,024 tokens have to be identical. After that first block, further hits accrue for every 128 additional identical tokens.

One consequence deserves its own sentence. A single character change inside the cached prefix is a cache miss, and everything behind that character gets reprocessed at full price. Azure states it directly: one character difference in the first 1,024 tokens gives you a cached_tokens value of zero.

The prefix is assembled in a fixed order. Anthropic renders tools first, then system, then messages, and documents that a change at one level invalidates that level and every level after it. Edit one tool definition and the system prompt and the whole conversation history behind it drop out of the cache with it. Switching models resets you as well, because cache entries belong to a model.

Google tells Gemini developers the same thing from the other end: put large and common content at the beginning of the prompt.

Cache tokens on your invoice

Cached tokens come in two flavours and they are priced differently. A cache write is the request that stores the prefix. A cache read is every later request that gets to reuse it. Anthropic reports both separately in the response, next to the tokens that were not cached at all:

"usage": {
  "input_tokens": 40,
  "cache_creation_input_tokens": 15000,
  "cache_read_input_tokens": 0
}

On the next call with the same prefix those last two numbers swap places, and that swap is the whole point. OpenAI and Azure expose a single cached_tokens field inside prompt_tokens_details instead, which counts reads.

Anthropic publishes the multipliers against its base input price: a write costs 1.25 times base for the default five-minute cache and 2 times base for the one-hour variant, while a read costs 0.1 times base. Run the arithmetic on the five-minute cache and it pays for itself on the second request: 1.25 plus 0.1 is 1.35 versus 2.0 for two uncached calls. The one-hour cache needs a third request before it wins, since 2.0 plus 0.1 is worse than two uncached calls but 2.2 beats three.

Azure's model is different again and has shifted over time. Models before gpt-5.6 charge nothing to write to the cache; from gpt-5.6 onward writes are billed on top of the discounted reads. Check the current pricing page before you build a business case on any of these numbers, because this is the part of the feature that moves.

Ordering the prompt is a design decision

Two teams can ship the same feature, send the same tokens, and get invoices that differ by a factor of five. The difference is usually the order.

Here is the failure that catches almost everyone. Your assistant has a large fixed instruction block, and someone adds a helpful line at the top: Current time: 2026-07-20 14:32:07. That line sits in front of everything else, so the prefix hash changes on every request. You now write a fresh cache entry every call, pay the write premium every call, and never read a single one. The feature is switched on, the invoice is worse than if it were off, and nothing in the response says so.

The fix is an ordering rule you can apply to any provider:

1. Frozen content first. The instructions, tool definitions and reference documents that are byte-identical across every request in the system.

2. Session-stable content next. Things that hold still for one user or one conversation, like a retrieved account file.

3. Volatile content last. Timestamps, request IDs, the user's actual question. Azure spells out the same order: stable content such as system instructions and examples at the start, variable content at the end.

Serialisation counts as ordering too. A dictionary dumped to JSON without sorted keys, or a set iterated in whatever order it happens to yield, produces different bytes on different runs even when the content is identical. The cache sees a different prefix and misses.

What to watch out for with prompt caching

The entry expires faster than you think. Azure's in-memory cache typically clears within five to ten minutes of inactivity and is always gone within an hour. Anthropic's default entry lives five minutes and is refreshed for free every time it is used. Longer retention exists on both sides, up to an hour on Anthropic and up to 24 hours with Azure's extended retention. A workload with a request every thirty seconds keeps its own cache warm; a workload with one request an hour caches nothing useful and pays the write premium every time.

Anthropic states the refresh rule plainly: "By default, the cache has a 5-minute lifetime. The cache is refreshed for no additional cost each time the cached content is used."

Verify instead of assuming. The cache fields in the response are the only honest signal. If the read counter stays at zero across repeated calls that should share a prefix, something in front is changing. Diff the rendered prompt bytes between two requests and you will find it.

Personalisation fragments the cache. Interpolating a customer name or account ID into the system prompt gives every user their own prefix and kills sharing across users. Move it behind the fixed block instead.

Short prompts do not cache at all. The minimum is roughly a thousand tokens on OpenAI and Azure and a couple of thousand on recent Gemini models, and a prompt below the threshold is simply processed without caching. No error is returned. If your prompt genuinely starts differently on every call, there is no reusable prefix and caching is not the lever you are looking for.

How the providers differ

The mechanism is shared; the controls are not.

Anthropic gives you explicit control. You mark cache breakpoints with a cache_control field on a content block, up to four per request, and the cache runs up to the marked point. That precision is useful and it is also the sharp edge: put the breakpoint after a block that contains a timestamp and you pay for a write on every request without ever getting a read. The breakpoint belongs on the last block that stays identical across the requests you want to share a cache.

OpenAI and Azure OpenAI cache automatically for eligible requests with no code change and no opt-out. Newer models accept a prompt_cache_key that you reuse across requests sharing a prefix, which improves matching. Azure notes that if requests for the same prefix and key exceed roughly 15 per minute some will miss, so high-volume workloads should spread across several keys while keeping each key mapped to a stable prefix.

Google Gemini runs implicit caching by default on its recent models and also offers explicit caches you create and manage yourself. Minimum sizes differ per model, in the low thousands of tokens.