Non-determinism (LLM output)

What is non-determinism in LLM output?

Non-determinism is what you have when the same input to a system does not reliably produce the same output. Send an identical prompt to a large language model twice, with identical settings, and you can get two different answers back. Nothing broke. That is simply how the system behaves.

Part of that variation is deliberate. The model produces a score for every possible next token, and a sampling step picks one from that distribution. Two settings shape the pick: temperature, which sharpens or flattens the distribution, and top_p, which trims its tail. Turn the randomness down and the variation shrinks.

The rest is not deliberate at all, and that is the part that catches teams out. It comes from the hardware the model runs on rather than from the sampling settings, and there is no API parameter that switches it off.

Why temperature 0 is not a guarantee

At temperature 0, sampling collapses to greedy decoding: at every step the model takes the highest-scoring token, no dice are rolled, and the output should therefore be fixed. Vendors say otherwise. Anthropic's API reference states that even with a temperature of 0.0, the results will not be fully deterministic.

The first reason is arithmetic. Floating-point addition is not associative, so (a + b) + c and a + (b + c) can differ in the last bits. A GPU kernel splits a large sum across many threads, and the order in which those partial results get combined changes those last bits. Usually that is invisible. When two candidate tokens sit very close together in score, it flips which one wins, and from that token onward the two answers walk away from each other.

The second reason sits underneath the first: the batch. A hosted model serves many customers at once, and your request is packed together with whatever else arrived in the same moment. Work published by Thinking Machines Lab puts the blame there. The kernels behind matrix multiplication, normalisation and attention are not batch invariant, so the same request computed inside a batch of four and inside a batch of forty comes out with slightly different numbers. How busy the server is never appears in your request, yet it changes your result.

Their measurement is worth remembering. Asking one open model the same question a thousand times at temperature 0 returned 80 distinct completions, and the first divergence showed up around the hundredth token. Short answers often look perfectly stable. Long ones drift.

What a seed fixes and what it does not

OpenAI and Azure OpenAI accept a seed parameter. Pass the same integer with the same prompt and the same settings, and the service makes a best effort to sample deterministically. Not every provider offers one; Anthropic's Messages API has no seed parameter.

Best effort is the operative phrase, and the vendors are candid about it. Microsoft's documentation for Azure OpenAI says that determinism isn't guaranteed, that variability still turns up even when the seed and the system_fingerprint match, and that longer responses are generally less deterministic than short ones. The system_fingerprint field exists so you can tell when the backend configuration changed underneath you, which is a fair signal that a difference in output was not your fault.

Microsoft's own worked example is honest about the limit. Three calls with seed=42, temperature 0.7 and a 50-token cap produced two byte-identical stories and a third that tracked the others for one sentence and then went its own way.

An LLM call next to a SQL query

This is the shift that matters for anyone who has spent a career in reporting. Run a SQL query against an unchanged table and you get the same rows, in the same order if you asked for an order, today and next Tuesday. That property is what makes a reconciliation possible: if the number moved, either the data changed or the query changed, and both are things you can inspect.

An LLM call breaks that assumption. The output can move while the prompt, the model and the data all stayed still. So the question "why is this different from last week?" no longer has a clean answer, and neither does "can you prove this is what the system produced on 3 March?".

It also breaks the usual test. A test that asserts exact string equality against a golden answer will pass in your first run and fail in the fifth, and the failure tells you nothing. This is why evals score a range of acceptable behaviour rather than checking for one exact string.

What to watch out for when an LLM sits in a pipeline

• Keep the non-deterministic step out of the number. Use the model for classifying, extracting or drafting, and let deterministic code do the aggregating. A total that is recomputed by a model on every refresh cannot be reconciled.

• Log the whole call, not the answer. Prompt, model version, parameters, seed if you set one, system_fingerprint if you get one, and the raw response. Without that record you cannot reconstruct an output later, and an audit trail that stops at the answer will not survive a serious question.

• Constrain the output shape. Ask for a value from a fixed list or a JSON object against a schema. Variation in wording matters much less once the field that lands in your warehouse can only take one of six values.

• Test on distributions, not on single runs. Run the same case fifty times and look at how often you get an acceptable answer. One green run proves very little.

• Watch out for silent backend changes. A provider can update the serving stack without touching the model name, so pin versions where the API lets you and re-run your evals when the fingerprint moves.