Temperature: why a chatbot answers differently each time

Why does a chatbot answer differently each time?

Ask a chatbot the same question twice and you often get two different answers. Nothing is broken. The program is not looking up a stored reply. It is choosing its reply one small piece at a time, and the choice involves a controlled amount of chance. The dial that controls how much chance is called temperature.

Step 1: the model scores every possible next word

A language model writes by repeating one job: given the text so far, work out what could come next. (Real models work with small pieces of words called tokens; we will say "words" here to keep things simple.) For the text The cat sat on the, the model gives every candidate a score. A higher score means "fits better". These raw scores are called logits.

The scores below are made up for this lesson, so you can see the mechanism with numbers small enough to read. A real model scores tens of thousands of candidates, but the steps are the same.

Step 2: turn scores into probabilities

Scores are awkward to use. They can be negative, and they do not add up to anything. So the model converts them into probabilities: numbers between 0 and 1 that add up to 1, like slices of a pie. The standard recipe is called softmax. For each score, compute e raised to that score, then divide by the total of all of them. Bigger scores get bigger slices, and no slice is ever exactly zero.

Temperature enters here. Before the recipe, every score is divided by the temperature T:

import math

words  = ["mat", "sofa", "floor", "roof", "moon"]
logits = [4.0, 3.0, 2.5, 1.0, -1.0]

def softmax(scores, temperature=1.0):
    scaled = [s / temperature for s in scores]
    biggest = max(scaled)                    # subtract the max so exp() cannot overflow
    exps = [math.exp(s - biggest) for s in scaled]
    total = sum(exps)
    return [e / total for e in exps]

for t in (0.5, 1.0, 2.0):
    probs = softmax(logits, t)
    row = "  ".join(f"{w} {p:.1%}" for w, p in zip(words, probs))
    print(f"T={t}: {row}")

Running it prints:

T=0.5: mat 84.2%  sofa 11.4%  floor 4.2%  roof 0.2%  moon 0.0%
T=1.0: mat 60.7%  sofa 22.3%  floor 13.5%  roof 3.0%  moon 0.4%
T=2.0: mat 41.9%  sofa 25.4%  floor 19.8%  roof 9.4%  moon 3.4%

The same five scores, three different pies. Dividing by a small number (0.5) stretches the gaps between scores, so the leader takes most of the pie. Dividing by a big number (2.0) squeezes the gaps, so the pie is shared more evenly. The score order never changes: mat is always on top. Only how confident the pie is changes.

Try it: turn the dial and draw words

Try this: set the dial to 0.1 and draw 100 times. Every draw, or very nearly every one, is mat. Move it to 2.0 and draw again: sofa and floor show up often, and now and then even moon sneaks in. Each draw is like spinning a wheel where the slices are the probabilities you see.

Step 3: pick one, then repeat

Once the model has a pie, it spins the wheel and takes the word it lands on. That word is added to the text, and the whole process starts again for the next word. A reply of 100 words is 100 spins. One different early choice (sofa instead of mat) changes everything the model reads afterwards, so two replies can drift far apart even when each single spin looked harmless.

What happens at the extremes

As the temperature gets close to 0, the top word takes nearly the whole pie. Many tools treat a setting of 0 as "always take the highest-scoring word", often called greedy choice. You get the most predictable reply, which suits tasks with one correct answer, like extracting a date from a message. The cost is that it can be repetitive and bland.

At high temperature, the pie flattens and unlikely words get real chances. That can make writing feel more varied and surprising, which suits brainstorming names or story ideas. It also makes odd, off-topic or wrong choices more likely. Very high values can produce nonsense, because the unlikely words keep getting picked and each one pulls the next choice further from sense.

What temperature does not do

It is easy to hear "higher temperature means more creative" and "lower temperature means more correct". Only the first half of the second idea is safe. Low temperature makes the model consistent, not right. If the model's top-scoring word is a wrong fact, a low temperature will repeat that wrong fact every time. Temperature changes how the model chooses among its own scores. It does not add knowledge, and it does not check anything.

So what for your own code?

If you use a chatbot through a website or a programming interface, look for a temperature setting. The range and the default differ between products, so read the documentation of the one you use instead of assuming. A sensible habit is to lower it when you want repeatable answers (data extraction, classification) and raise it when you want variety (naming, ideas). Whatever you pick, still check important facts yourself.

And if you ever write a simulation, a game or a shuffler, the pattern you just saw is useful far beyond AI: turn scores into probabilities, then draw from them. A weighted random choice is the same idea wherever it appears.