Token streaming (LLM output)

What is token streaming?

Token streaming means a language model sends its answer while it is still writing it. Every time it finishes a few tokens, the API pushes them to your application over a connection that stays open, and your application shows them straight away. Without streaming, the API waits for the last token and returns everything at once. The model works the same either way, producing one token at a time as the Tokens entry explains. Streaming only changes when you get to see them.

The usual transport is server-sent events, an ordinary HTTP response with the content type text/event-stream that stays open and delivers one small event at a time. Anthropic, OpenAI and Google all work this way: you set a stream flag on the request and get a series of events instead of one JSON document. Google's documentation puts the default plainly: the model returns a response only after the entire generation process is complete, and streaming exists for more fluid interactions.

This is not the same thing as streaming data. That term covers a pipeline that processes business events such as orders or payments continuously as they arrive. Token streaming is about delivering one model answer to one caller.

Streaming versus a single response

Take a 500-token answer, roughly 375 words, from a model producing 50 tokens per second. Generating it takes about ten seconds either way. Without streaming, the user watches a spinner for ten seconds and then starts reading. With streaming, the first words appear as soon as the model has read the prompt, and the rest arrives while the user is already reading, so the answer is mostly read by the time it finishes.

Two numbers describe that. Time to first token is how long the user waits before anything appears. Tokens per second is how fast text keeps coming after that. A chat product lives on the first number; the second only has to stay above reading speed. OpenAI's latency guide calls streaming the single most effective approach for making users wait less, because there is a difference between waiting and watching progress happen.

For the developer it runs the other way. A single response is one HTTP call and one JSON object. A stream is a sequence of events you read, assemble and render while they keep coming.

What changes for the developer

You handle events, not a result. Anthropic's API sends a message_start, then per content block a start, a series of deltas and a stop, then a message_delta with the stop reason and a cumulative token count, then message_stop. OpenAI's Responses API has the same shape under different names.

Structured output arrives as partial JSON. Each delta is a fragment of a JSON string, not valid JSON on its own. Either you wait for the closing event and parse once, which is what Anthropic's Java SDK documentation tells you to do, or you use a parser that handles partial JSON.

Tool calls arrive mid-stream. Tool input comes in as partial JSON too. Anthropic notes that current models emit one complete key and value at a time, so there may be delays between streaming events while the model is working. You cannot run the tool until the block closes, which OpenAI signals with a response.function_call_arguments.done event.

Stopping and restarting are yours to build. The user can hit stop halfway, and the tokens generated up to then still count as output on the bill. A dropped connection is worse: after 300 tokens you cannot quietly retry, because the user has read those 300 tokens, so you either restart and replace what is on screen or ask for a continuation. Errors also arrive inside the stream, where Anthropic sends an overloaded_error event instead of the HTTP 529 a normal call would return.

When streaming pays off

Anywhere a person sits waiting for text, a chatbot, a drafting assistant in a mail client, an AI explaining a report, the product feels slow without streaming and quick with it. A voice agent cannot work without it at all, because the text-to-speech step needs the first sentence before it can start speaking. Google's Live API takes that further and streams audio in both directions over a WebSocket for low-latency voice.

A batch job gains nothing. Classify 20,000 invoices overnight and nobody is watching, so only total throughput counts. Anthropic's SDKs do require streaming for requests with a large max_tokens value, to avoid HTTP timeouts, but they hand you the assembled message anyway.

Streaming does not change what you pay. The bill is identical whether the tokens arrive in one block or in fifty events, because they are the same tokens. It changes the waiting time the user experiences, which is worth a lot with a person on the other end and nothing at all without one.

What to watch out for with token streaming

Moderation runs after the fact. OpenAI's documentation says streaming the output in a production application makes the content more difficult to moderate, and that moderation scores arrive after the full output rather than with the partial deltas. If a guardrail has to clear an answer before the customer sees it, you either buffer the stream and lose the effect, or accept that a wrong answer appears word by word and gets withdrawn afterwards.

People read half an answer and act on it. A model often opens with a number and qualifies it two sentences later. In a stream the user reads the number, stops and picks up the phone. Mark the end of an answer clearly, and instruct the model to give its caveats before its conclusion, not after.

Partial output in your logs. When the user cancels or the connection drops, decide what lands in your logs and in the conversation history. Store the text with a flag saying it was cut off, or the next turn treats a truncated answer as a finished one.