Temperature (LLM parameter)

What is temperature in an LLM?

Temperature is a sampling setting that controls how much randomness a large language model uses when it chooses the next token. At each step the model produces a raw score, called a logit, for every token in its vocabulary. A softmax function turns those scores into probabilities, and temperature scales the logits before that softmax runs.

A low temperature divides the logits by a small number, which widens the gaps between them and sharpens the distribution toward the most likely token. A high temperature flattens the distribution instead, so less likely tokens get a real chance of being picked.

Temperature does not make the model smarter. It adds no facts and does not make the model more creative in any deep sense; it only changes which tokens get sampled. If a low-temperature answer is wrong, a higher temperature will not fix it, only change how it is wrong.

Temperature 0 and determinism

Set temperature to 0 and sampling collapses to greedy decoding: at every step the model takes the highest-probability token and no randomness is left. Ask the same question twice and you will usually get the same answer, so people reach for 0 when they want repeatable output.

Usually is the honest word. Even at temperature 0, most hosted models are not guaranteed to return the exact same text every time. Requests get batched on the GPU, and floating-point arithmetic is not perfectly associative, so small numerical differences can flip which token wins a close call. Anthropic says as much in its API docs: even at a temperature of 0.0, results will not be fully deterministic. Treat 0 as near-deterministic, not a guarantee.

Which value for which task

Keep temperature low, roughly 0 to 0.3, for anything you are going to parse or check:

• Extraction and classification. Pulling fields out of a document or sorting tickets into categories, where one correct answer exists.

• SQL and structured output. Generating a query for NL2SQL, or JSON that has to fit a schema, where a stray token breaks the parser downstream.

• Grounded answers. Question answering over retrieved context, where the model should stay with what it was given.

Raise it, roughly 0.7 and up, when variety is the point: brainstorming, alternative headlines, or copy variants. A higher temperature also makes hallucination more likely, so keep it to tasks where a person reads the output, not a pipeline that trusts it. Good prompt engineering matters more than the exact number, so start at the model's default and change temperature only when you have a measured reason.

Temperature versus top_p

Top_p, or nucleus sampling, is the other sampling knob. It keeps the smallest set of tokens whose probabilities add up to p and samples from that set. Temperature reshapes the whole distribution; top_p trims its tail. OpenAI says to change temperature or top_p but not both, and Anthropic treats top_p as advanced, so tune only one; two overlapping controls make the behaviour hard to reason about.

The ranges also differ by vendor, so the numbers are not portable. Claude takes a temperature from 0.0 to 1.0 and defaults to 1.0, while OpenAI's models take 0 to 2 and default to 1. A prompt set to temperature 1.0 on one provider will not behave the same on the other.