Stateless (LLM API)

What is a stateless LLM API?

A language model API is stateless: it remembers nothing about you between one call and the next. Every request arrives with no history attached, the model reads whatever text is in that request, produces an answer, and forgets the whole thing. Send the identical request twice and the second one starts as cold as the first.

That sits oddly next to how a chat assistant feels. You ask a follow-up question, it clearly knows what you were talking about a minute ago. The model did not remember that. The application in front of it took the entire conversation so far and pasted it back in as part of the new request.

Anthropic's Messages API documentation puts the design plainly: the API is for "single queries or stateless multi-turn conversations", and you "specify the prior conversational turns with the messages parameter" on every new call. The conversation is an argument you pass, not a thing the model holds.

Where the memory in a chat session actually lives

The memory lives in your application, and it travels over the wire on every turn. A three-turn chat sends something shaped like this:

[
  {"role": "user", "content": "How many customers churned in Q1?"},
  {"role": "assistant", "content": "Forty-two, mostly on the starter plan."},
  {"role": "user", "content": "And what did that cost us?"}
]

The last line is the only thing new. The first two are there so the model can work out what "that" refers to. On turn four, all three earlier messages get sent again, plus the new one. On turn thirty, twenty-nine get sent again.

This is also why the context window is a hard ceiling on conversation length rather than on message length. Every turn you add is permanent freight for every turn after it, until the whole conversation no longer fits and the oldest messages have to be dropped or summarised.

Why the bill grows faster than the usage

Because you resend the history, you get billed for the history. OpenAI's conversation state guide states the consequence without hedging: "Even when using previous_response_id, all previous input tokens for responses in the chain are billed as input tokens in the API."

Work it through with round numbers. Say you have a 2,000-token system prompt, and each turn adds about 500 tokens of question plus answer.

• Turn 1 sends 2,000 input tokens.

• Turn 2 sends 2,500.

• Turn 20 sends 11,500, more than five times what the first turn sent.

Across those twenty turns you are billed roughly 135,000 input tokens, while only about 10,000 tokens of genuinely new text ever passed between you and the model. The gap is the same history being paid for again and again.

The practical shape of this: doubling the length of a conversation moves you closer to four times the cost than to two. When a finance team asks why the AI bill grew faster than the number of questions people asked, this is almost always the answer. Nobody used the tool more than expected. They used it in longer sessions.

Server-side state: what the Responses API changes

Newer APIs will hold the conversation for you. OpenAI's Responses API lets you "chain responses across turns by passing the previous response ID" through previous_response_id, and its Conversations API "works with the Responses API to persist conversation state as a long-running object with its own durable identifier". Azure OpenAI supports the same chaining and retains response data for 30 days by default.

This is worth being precise about, because it is easy to read as a fix for the cost problem. It is not. It moves the bookkeeping to the server, so your application stops shipping the transcript around and stops having to store it. The model underneath still reads the whole conversation on every turn, and, per the sentence quoted above, you still pay for all of it.

Server-side state is also optional rather than a new default. Azure's documentation notes that setting store=false gives you a stateless request while keeping the rest of the Responses API. What genuinely reduces the token count is compaction: Azure exposes a context_management option with a compact_threshold, which replaces older turns with a condensed summary once the conversation crosses that size.

How prompt caching changes the arithmetic

Prompt caching attacks the price of the resent tokens rather than their number. Anthropic describes it as "resuming from specific prefixes in your prompts", so a stable prefix (a long system prompt, a document, the earlier part of a conversation) is not reprocessed from scratch on the next call.

The pricing is where it earns its place. On Anthropic's API, a cache write costs 1.25 times the base input rate and a cache read costs 0.1 times it, with a default five minute lifetime that refreshes for free each time the cache is used. A conversation held at a steady pace can therefore keep paying a tenth of the list price for its own history.

Caching and statelessness are not in tension. Caching is what makes statelessness affordable: the API still receives the entire conversation on every call, it just charges you far less for the part it has seen recently.

What to watch out for with stateless APIs

Latency climbs with the transcript, not with the question. A one-line follow-up at turn thirty carries thousands of tokens of history in front of it, and the model reads all of them before it starts answering. A chat that felt instant in the morning feels sluggish by mid-afternoon for that reason alone.

Answer quality drifts in long sessions. Models recall the start and end of a long prompt better than the middle, so a constraint you set fifty turns ago competes with everything piled on top of it. Starting a fresh conversation is often a genuine fix rather than a workaround.

Per-user cost is unpredictable if you only count messages. Two people sending the same number of messages can differ by an order of magnitude in spend, depending on whether they work in short bursts or one endless thread. Bill and monitor on tokens.

Trimming history is a design decision, not a detail. Dropping the oldest messages is cheap and silently loses instructions. Summarising is dearer and loses specifics. Whichever you pick, decide it deliberately and tell the model what it no longer has.