In Production · AI · Fundamentals
We’re bad intuitive statisticians. So are LLMs.
A hostel in Foz do Iguaçu, three doors and a goat taught me more about temperature and top-p than any model card.
I was backpacking through Foz do Iguaçu with Kahneman’s Thinking, Fast and Slow in my bag when two other backpackers asked what it was about. I gave them the book’s most famous trap: a man is described as shy and helpful. Is he a librarian or a farmer?
They both said librarian. Everyone does. But there are far more farmers than librarians, so the boring answer is the likely one. The description wins and the base rate loses. We are not intuitive statisticians, and the people who build with language models are no exception.
A model does not look an answer up. At every step it holds a probability distribution over the next token and draws from it. Temperature and top-p do not make it more creative or more careful; they change the shape of that distribution before the draw. Most teams tune them the way my friends in the hostel guessed: by feel, from one demo.
Loading interactive figure…
Static version: at temperature 0.8 and top-p 0.9, “tests” is drawn 44% of the time; DNS, intern and moon are cut.
Drag temperature down to 0.1 and “tests” wins every single time: deterministic and confidently dull. Push it to 2 and “moon” starts turning up in your incident summary. Top-p is the guard rail. It cuts the tail before the draw, so a hot temperature cannot reach tokens that were never plausible. None of this shows in a single answer. It shows when you sample a thousand times, which is exactly what your users will do for you.
Three doors
Back in the hostel I pulled out Monty Hall. Three doors, one car, two goats. You pick a door. The host, who knows where the car is, opens another one and shows you a goat, then offers a switch. Both of them said it made no difference: two doors left, fifty-fifty. Play a round, then let the machine play a thousand.
Loading interactive figure…
Static version: staying wins 1 in 3; switching wins 2 in 3.
Switching wins two times in three. Your first pick is right a third of the time, so it is wrong two thirds of the time, and the host has just shown you where not to look. His choice is evidence, and evidence changes the distribution. That is Bayes, and it is also the whole job with language models: the prompt, the retrieved documents and the examples are evidence that move what the model samples from. Temperature only decides how far into that distribution you let it reach.
There was one more trap that night, and it is the one I now see most in evaluations. A Fluminense striker scores 1 of 3 shots in the league and 95 of 97 in the Libertadores. A Flamengo striker is better in both: 101 of 300 and 96 of 97. Add the totals and Fluminense wins, 96% to 50%. Simpson’s paradox. A model can beat the baseline in every slice of your eval and still lose overall, or the other way around.
So I pick temperature per feature, from samples. I judge prompt changes on hundreds of runs, never on the one answer from the demo. And I split every eval by segment before I believe its total.