What Is Temperature in AI? A Beginner's Guide, Measured on a Real Model
Temperature is the setting that decides how adventurous a language model is when it picks its next word. At temperature 0 it always takes the most likely word, so the same question gets the same answer, almost always. As the temperature rises, less likely words get a real chance, and the answers start to vary, then to wander, and at some point to fall apart.
That is the whole idea, and it is easier to see than to describe. So I asked a small model, Llama 3.2 3B running locally through llama.cpp on a MacBook, the same question 100 times at four temperatures: "Suggest a name for a new coffee shop." Every other setting that limits the choice was switched off, so only the temperature changed; what those other settings do comes at the end.
| Temperature | Different answers out of 100 | The most common answer |
|---|---|---|
| 0 | 1 | "Brewed Awakening", 100 times |
| 0.7 | 12 | "Brewed Awakening", 55 times |
| 1.0 | 37 | "Brewed Awakening", 33 times |
| 1.5 | 99 | "Brewed Awakening", 2 times |
At 0 the model is a vending machine. At 0.7 it has a favourite and a few alternatives. At 1.5 almost every answer is new, and more than a quarter of them are not names at all but strings that mix several alphabets. The rest of this article explains why, with the model's own numbers.
How a model picks the next word
A language model writes one token at a time. A token is a word or a piece of one: "Apple" is one token, "Banana" starts with the token "Ban". Before each token, the model gives every token it knows a score, and turns those scores into probabilities that add up to 100%. Then one token is picked, added to the text, and the whole thing repeats.
Here is what Llama 3.2 3B thinks the first word of its answer should be, when asked to "Name one fruit. Answer with one word.", at temperature 1:
| Token | Probability |
|---|---|
| Apple | 63.7% |
| Orange | 15.4% |
| Ban (Banana) | 8.2% |
| M (Mango) | 5.2% |
| Str (Strawberry) | 4.1% |
| P (Pineapple) | 2.1% |
How the token is picked from that list is the part temperature controls. The simplest rule, called greedy decoding, always takes the top one: Apple, every time. The other rule, sampling, rolls a die weighted by the probabilities: Apple about two times in three, Orange about one time in seven, and so on. Sampling is what makes a chatbot's answers vary from one try to the next.
The die does what the table says. I let the model answer the fruit question 300 times by sampling at temperature 1: Apple came up 185 times (62%), Orange 47, Banana 28, Mango and Strawberry 16 each, Pineapple 7 and Pear once. That is also where the names in brackets come from: in those 300 answers, "M" always went on to Mango and "P" to Pineapple. Pear has a token of its own further down the list.
What temperature does
Temperature changes the probabilities before the die is rolled. A low temperature makes the favourite even more of a favourite; a high temperature flattens the list, so the long tail of unlikely tokens gets a bigger share. Here is the same fruit question at three temperatures:
| Token | Temperature 0.5 | Temperature 1.0 | Temperature 1.5 |
|---|---|---|---|
| Apple | 92.0% | 63.7% | 39.8% |
| Orange | 5.4% | 15.4% | 15.4% |
| Ban | 1.5% | 8.2% | 10.2% |
| M | 0.6% | 5.2% | 7.5% |
| Str | 0.4% | 4.1% | 6.4% |
| P | 0.1% | 2.1% | 4.2% |
The mechanism is one division. The model's raw scores, called logits, are divided by the temperature, and then turned into probabilities with a function called softmax, built on the number e (about 2.718) raised to the power of each score:
In words: the probability of a token is e raised to its score divided by T, divided by the same thing added up over all tokens. Here pi is the probability of token i, zi is its score, T is the temperature, and the sum at the bottom runs over every token the model knows, which makes all the probabilities add up to 100%.
Dividing by a number below 1 stretches the gaps between scores, and the exponential turns bigger gaps into much bigger ratios, so the leader pulls away. Dividing by a number above 1 squeezes the gaps, so the list flattens. At temperature 1 nothing changes: the probabilities are the model's own.
You can check this against the table. Halving the temperature from 1 to 0.5 doubles every score, which squares every probability before they are scaled back to add up to 100%. Apple's 63.7% squared is 0.406 and Orange's 15.4% squared is 0.024; with the smaller tokens added, the squares sum to about 0.44, and 0.406 divided by 0.44 is 92%, the number the model reported. The temperature did not teach the model anything about fruit. It only changed how far the model trusts its own first choice.
Temperature 0 cannot be a division, so tools treat it as "always take the top token", which is greedy decoding. The 100 identical answers below are what that looks like.
100 coffee shops at four temperatures
A one-word answer hides how quickly the choices add up. A coffee shop name is several tokens, and each one is a new roll of the die, so a small change in temperature has many chances to matter:
At 0, all 100 answers were "Brewed Awakening". Same question, same answer, which is what you want from a step that has to be repeatable.
At 0.7, "Brewed Awakening" still won 55 times, "Brewhaven" came second with 25, and ten other names shared the rest. That is close to where many tools start by default (llama.cpp's server uses 0.8): varied, but still sensible.
At 1.0, the model's own probabilities, there were 37 different names. The favourite won only a third of the time, and the long tail showed up: "Brewcraft Co.", "Brew & Co.", "Brew Haus".
At 1.5, 99 of the 100 answers were different, and many stopped being coffee shop names. 28 contained characters from other alphabets, 11 ran on past 40 characters, and more had stray code in them: "BrewhavenCppMethod", ".geomDataContext", "Brewchesterанный". A flattened list gives the thousands of tokens that make no sense here a combined share big enough to be picked, and once one odd token is in the text, the next ones follow from it.
What temperature to use
The measurements above are from one small model. The ranges below are my starting points, not a measurement, and the right value for your task is the one you check on your own examples.
| Task | Starting point | Why |
|---|---|---|
| Extracting fields, classifying, routing, JSON | 0 | There is one right answer; variety is only a source of errors. |
| Answering questions, writing code, summaries | 0 to 0.5 | Mostly one right answer, with room for phrasing. |
| Conversation, rewriting, drafting text | 0.5 to 0.8 | Natural variety without losing the thread. |
| Brainstorming names, ideas, alternatives | 0.8 to 1.0 | You want the long tail, but in readable words. |
| Anything above 1 | rarely | Without a top-p or top-k cut, the coffee shops above show what the flattened tail brings in. |
Two things to keep in mind. First, temperature is not a creativity dial that makes a model smarter; it only moves probability between answers the model already had. A better brainstorm usually comes from a better question, or from asking for ten names at once, not from a higher temperature. Second, the default is often not 0: llama.cpp's server defaults to 0.8, and other tools have defaults of their own, so if a step needs the same answer every time, set the temperature yourself.
Temperature 0 is not a promise
Temperature 0 makes a model pick the top token, but it does not make the model's arithmetic identical everywhere. In a test of small models at the edge, the same model file at temperature 0 gave a different answer on 23 of 100 tickets when it moved from a Mac's GPU to a thin client's CPU, and a different answer on 2 of 200 tickets when the same request was simply sent twice, because the server reused a cached prompt. When two tokens are almost tied, a rounding difference is enough to change which one is on top.
Cloud APIs differ too. According to Anthropic's documentation, the API version its current SDKs use rejects the temperature, top_p and top_k parameters on Claude models, and the same request sent twice to Claude Haiku 5.5 in that test changed its answer on 3 of 200 tickets. If a workflow needs the same answer for the same input, store the first answer and reuse it, rather than relying on temperature 0 to produce it again.
Temperature's neighbours: top-k and top-p
Temperature rarely works alone. Most tools also cut the list before the die is rolled:
- top-k keeps only the k most likely tokens. With k = 40, token number 41 can never be picked, however flat the temperature makes the list.
- top-p, also called nucleus sampling, keeps the smallest set of top tokens whose probabilities add up to at least p. With p = 0.95, the tail that shares the last 5% is dropped.
- min-p drops every token less likely than a fraction of the top one.
For the measurements above I switched all three off (top_k: 0, top_p: 1, min_p: 0), so temperature was the only setting that changed anything. llama.cpp's server has all three on by default (top-k 40, top-p 0.95, min-p 0.05), so I asked the coffee shop question 100 more times at each of the two higher temperatures with those defaults:
| Temperature | Filters off: different answers | Default filters: different answers | Default filters: broken answers |
|---|---|---|---|
| 1.0 | 37 | 17 | 0 |
| 1.5 | 99 | 20 | 0 |
With the defaults, temperature 1.5 behaved much like 1.0: "Brewed Awakening" won 39 times at both, and not one answer fell apart. The cuts remove the tail before the temperature can promote it, which is why the same temperature can be harmless in one tool and wild in another. A practical rule: change temperature or top-p, not both at once, so you know which one made the difference.
How this was measured
Llama 3.2 3B Instruct, the Q4_K_M file from bartowski/Llama-3.2-3B-Instruct-GGUF on Hugging Face, served by llama.cpp b12074 on a MacBook Pro with an M1 Pro. Each request was sent to the server's /completion endpoint with the model's own chat template around the question, the prompt cache off, and top-k, top-p and min-p switched off. The probabilities are the server's own, read with n_probs and post_sampling_probs, which report them after temperature has been applied. The core of the script:
OFF = {"top_k": 0, "top_p": 1.0, "min_p": 0.0}
def samples() -> dict:
p = prompt("Suggest a name for a new coffee shop. Answer with the name only.")
out = {}
for t in [0.0, 0.7, 1.0, 1.5]:
answers = []
for _ in range(100):
r = post("/completion", {"prompt": p, "n_predict": 24, "temperature": t, "cache_prompt": False, **OFF})
answers.append(r["content"].strip())
out[str(t)] = answers
return outThe second coffee shop run, with the server's default filters, used the same requests without the OFF settings. This is one model, one question per experiment and 100 tries per temperature, so the exact counts would move with another seed or another model. I would expect the shape to hold: one answer at 0, a favourite with alternatives below 1, and a long tail of nonsense once the tail is allowed in. If you want to see it on your own model, the same two requests work on any llama.cpp server, and any API that accepts temperature should show the same pattern in the answers.
Temperature is one of the first settings to understand when you put a model into a real workflow. The next questions are how to tell whether its answers are right, which I wrote about in how to evaluate AI agents, and when you need a model at all, which is the subject of when to use RAG and when simple search is enough.