LLM Sampling Temperature
Summary: Temperature controls how randomly an LLM picks each next token from its predicted probabilities. Temperature 0 means "always take the top token" (greedy decoding): more consistent, but not more accurate and not perfectly deterministic.
At a glance​
| Question | Answer |
|---|---|
| What is it? | A sampling setting that sharpens or flattens the next-token probability distribution |
| Why does it matter? | It trades consistency against variety in every LLM call |
| When do I use it? | Low for classification, extraction, and structured output; higher for creative work |
How it works​
- At each step, the model scores every vocabulary token (logits).
- Each logit is divided by the temperature T, then converted to probabilities with softmax.
- A token is sampled from those probabilities, appended, and the loop repeats.
- Temperature changes only how the token is picked, not the model itself.
probability_i = exp(logit_i / T) / Σ exp(logit_j / T)
- T > 1: flattens the distribution, so unlikely tokens are chosen more often.
- T < 1: sharpens it toward the top choice.
- T = 0: would divide by zero, so providers treat it as greedy decoding (argmax).
Example​
Next token after "The capital of France is":
| Token | Logit | T = 2 | T = 1 | T = 0.5 | T = 0 |
|---|---|---|---|---|---|
| Paris | 2.0 | 48% | 63% | 84% | 100% |
| Lyon | 1.0 | 29% | 23% | 11% | 0% |
| Nice | 0.5 | 23% | 14% | 4% | 0% |
Why T = 0 is not fully deterministic​
- GPU floating-point arithmetic is not perfectly associative. Batching, hardware, and kernel choices can flip near-tied tokens.
- One different token changes everything generated after it.
- Providers change serving infrastructure over time.
Choosing a temperature​
| Task | Typical temperature |
|---|---|
| Classification, extraction, routing | 0 or near 0 |
| Structured/JSON output, code | Low (0–0.3) |
| General Q&A, summaries | Moderate (~0.3–0.7) |
| Brainstorming, creative writing | Higher (~0.7–1+) |
Related knobs: top_p samples only from the smallest set of tokens whose probabilities sum to p; top_k samples only from the k most likely tokens. Usually tune temperature or top_p, not both.
Remember​
- T = 0 makes the model consistently pick its top guess. A wrong top guess is consistently wrong.
- T = 0 does not prevent hallucination; a confident hallucination is still the top token.
- Greedy decoding can fall into repetition loops in long outputs.
- Low temperature reduces variance but does not replace validation, retries, or evals.
- Allowed ranges and support differ by provider and model. Check the model's docs.
Related notes​
- LLM Workflow App Practices (not yet published) - why workflow pipelines still need validation and evals even at low temperature