Inference cost

What is inference cost?

Inference cost is what you pay for running a model, every single time you run it. It is separate from what a model cost to train, and it is separate from what your project cost to build. It is the line that never stops.

That is the part that surprises people who come from classic software. A report you build once runs for years at the cost of the server underneath it. An AI feature you build once has a marginal cost per use, forever. So the question of whether something is worth automating changes shape. It is no longer only about the build effort, it is about the cost per run against the value of that run.

The unit is the token. Your prompt is counted in input tokens, the answer is counted in output tokens, and the bill is the sum of the two at different rates.

Where the bill actually comes from

Output costs several times more than input. Across providers, output tokens are priced at a multiple of input tokens, commonly around five times. That has a direct design consequence: a system that reads a lot and answers briefly is cheap, and a system that reads little and writes at length is not. If you are looking for savings, look at answer length before you look at prompt length.

Thinking is billed as output. Reasoning models produce internal reasoning tokens before they answer. You never see them, and you pay for them at the output rate. A model that reasons for a while before giving you three sentences can cost more than a model that writes three pages.

Tools add input tokens on every call. Every tool definition you attach, with its name, description and schema, goes into the prompt each time. An agent with twenty tools attached is paying for twenty tool descriptions on every single turn, whether it uses them or not.

Context accumulates in a conversation. Each turn resends the history. A twenty-turn conversation is not twenty small requests, it is a request that grows every turn.

The levers that actually work

Prompt caching. If a large part of your prompt is stable, a long system prompt, a policy document, a schema, you can have the provider cache it. Reading from that cache costs a fraction of the normal input price, typically around a tenth. Writing to the cache costs slightly more than normal input, which means caching pays for itself after very few reuses. For a support assistant that sends the same instructions on every request, this is usually the single biggest saving available.

Batch processing. If the work does not have to happen right now, providers offer batch processing at roughly half price on both input and output. Overnight classification of yesterday's tickets, enriching a table, generating descriptions in bulk: all good candidates.

Routing to a smaller model. Not every step needs your most capable model. Classification, extraction and routing are usually fine on a small model at a fraction of the cost, with the expensive model reserved for the step that genuinely needs judgement. This is often where half the bill goes.

Structured output. A model that returns JSON does not write an introduction or a closing paragraph. Fewer output tokens at the expensive rate, and easier processing afterwards.

Shorter answers by instruction. Telling the model how long the answer should be is a cost control, not just a style preference.

Why agents change the shape of the budget

The numbers that matter here come from Anthropic, who measured their own workload. A single agent uses roughly four times as many tokens as a plain chat conversation, because it loops, calls tools and reads back results. A multi-agent system uses roughly fifteen times as many tokens as a chat conversation.

Read those as an order of magnitude on their task rather than as a rate card for yours. What holds everywhere is the shape. A chat message is one request. An agent is an unknown number of requests, decided at runtime by the model rather than by you.

That has one practical consequence you should act on before you go live. A workflow with three fixed model calls has a bill you can predict. An agent that decides for itself how many calls to make does not, until you put a hard limit on the number of rounds and on the token budget. That limit is not a nice-to-have. It is the difference between a cost you can defend and a surprise.

What to watch out for with inference cost

The pilot never predicts the production bill. Five colleagues testing a tool produce a rounding error. Five hundred colleagues using it daily do not. Estimate on volume before you commit, not after.

Cost per correct answer is the number that matters. A cheaper model that gets it wrong more often costs you rework, and rework is more expensive than tokens. Compare on cost per usable result, not on price per million tokens.

Nobody owns the bill by default. When several applications call the same model deployment, the invoice arrives as one number. Putting an LLM gateway in front, with token metrics per consumer, is what turns that into a number per team.

Retries are invisible spend. An agent that fails and tries again pays twice. If your error handling is a simple retry loop, you are paying for every failure at full price.

Caching changes what a change costs you. Once you rely on prompt caching, editing your system prompt invalidates the cache. That is fine, but it means an innocent-looking wording change can move your bill for a day.