Image: Thinking Machines Lab
Ask a chatbot a question, copy the answer, then ask the exact same question again in a new chat. There is a good chance the second answer reads differently. The wording shifts, the examples change, and now and then the conclusion changes too. That can feel like a glitch, or like the AI is making things up as it goes. The truth is a mix of both design and engineering, and once it makes sense it becomes much easier to get answers that stay put.
The short version is that chatbots are built to pick their words with a little bit of chance, and even when that chance is switched off, the computers that run them can still nudge the result. Both parts are worth understanding, because they lead to different fixes.
A large language model writes one small piece of text at a time. Those pieces are called tokens, and a token is often a whole word or part of one. Google's Gemini documentation says a token is roughly four characters, and that 100 tokens come to about 60 to 80 words.
For every next token, the model does not produce a single answer. It produces a list of possible next tokens, each with a probability attached. After the words "The story of Thinking Machines Corporation begins not in a", a model might give "garage" a high chance, "boardroom" a small one and "laboratory" a smaller one still. The image above, from Thinking Machines Lab, shows exactly this kind of list.
Picking from that list is called sampling. As Thinking Machines Lab puts it in a detailed research post, getting a result from a language model involves turning its output into a probability distribution and then choosing a token according to those odds. That is the first and most obvious reason two answers differ. If the model is allowed to sometimes choose the second or third most likely word, then one different word early on can send the rest of the reply down a different path.
This is a feature, not a flaw. A bit of randomness is what keeps a chatbot from sounding robotic, and it is what makes it useful for brainstorming, writing and creative work. Ask for ten names for a new cafe and identical lists every time would not be much help.
Developers who use these models through an API can usually adjust how much randomness goes into that choice. The two best known controls are temperature and top-p.
Temperature is the main dial. Google's prompt design guide for Gemini describes it as the setting that controls the degree of randomness in token selection. Lower temperatures suit prompts that need a more predictable, less open ended answer, and higher temperatures can lead to more varied or creative results. A temperature of 0, according to the same guide, means the highest probability token is always selected.
Top-p works differently. Rather than changing how bold the model is, it trims the list of choices. The model only picks from the most likely tokens whose combined probability reaches a chosen cutoff, and everything below that line is ignored. Top-k is a similar idea that keeps a fixed number of top candidates. A simple way to picture it is that temperature decides how adventurous the model is, while top-p and top-k decide how big the menu is.
These dials are changing as models change. Google's guide notes that Gemini 4 and Gemini 3.6 and later models ignore custom values for temperature, top-p and top-k, return deprecation warnings, and will return errors in a future release. Anthropic's Messages API reference did not list temperature or top-p among its request options at the time of this writing, though it does offer an effort setting and a structured output format. On the newest models, then, randomness is increasingly something the provider manages rather than something the user tunes.
Most people never touch these settings in the regular chat apps, where the provider picks them. That is part of why the same question can return a different answer from one chat to the next.
If randomness is the cause, then turning the temperature down to 0 should give the same answer every time. In theory it does. In practice it often does not, and that gap is what researchers have been digging into.
A 2024 paper on arXiv titled Non-Determinism of "Deterministic" LLM Settings tested five language models set up to be deterministic on eight common tasks, running each one 10 times. The authors report accuracy swings of up to 15% across ordinary runs, and a gap of up to 70% between the best and worst possible results. They found that none of the models delivered repeatable accuracy across all tasks, much less identical text. The authors also wrote that people they shared early results with suggested the variation may be tied to how efficiently computing resources are used, and that the issue is not going away anytime soon.
So a setting labelled deterministic is not a promise. To see why, it helps to look at what happens inside the computers running the model.
Thinking Machines Lab published one of the clearest explanations in September 2025. The post, written by Horace He with others at the lab, starts from a fact about how computers handle decimal numbers. With floating point math, adding the same numbers in a different order can give a slightly different total. In one example from the post, summing the same short list of numbers in different orders produced 102 different results.
A common theory was that this happens because many processor cores work at the same time and finish in an unpredictable order. Thinking Machines Lab argues that this misses the main cause. Its post says the key issue is something called batch invariance.
Popular chatbots serve many people at once. To be efficient, the servers group requests from different users into batches and process them together. The size of that batch changes all the time depending on how busy the service is. According to Thinking Machines Lab, the math routines used to run the model can add numbers up in a different order depending on the batch size. The result is that the answer to one person's question can depend on how many other people happened to be using the service at that moment. No information leaks between users, the post stresses, but the server's load quietly changes the arithmetic.
Image: Thinking Machines Lab
Those differences are tiny, but a language model builds its reply one token at a time. If a tiny difference flips which token comes out on top at one step, every word after that can change.
The lab tested this on the Qwen3 235B model, asking "Tell me about Richard Feynman" 1,000 times at temperature 0. It got 80 different completions. All of them were identical for the first 102 tokens. Then 992 continued with Feynman being born in "Queens, New York" while 8 said "New York City".
The fix Thinking Machines Lab proposes is to rewrite the key building blocks of the model so they always add numbers in the same order, no matter the batch size. With these batch invariant routines switched on, the post says, all 1,000 completions came out identical. The lab released the code publicly and pointed to an example of running the vLLM inference engine in a deterministic mode.
There is a price. In the lab's own test with a smaller Qwen model on one GPU, a job that took 26 seconds with default settings took 55 seconds with the unoptimized deterministic version, and 42 seconds after an improved attention routine. The lab says it had not put much effort into speed yet and called the slowdown not disastrous. Even so, it shows why providers do not simply switch this on everywhere.
OpenAI has offered a partial tool of its own. Its cookbook guide on reproducible outputs explains a seed parameter for its Chat Completions API. Setting the same seed and keeping every other setting the same is meant to return mostly consistent outputs. OpenAI's wording is careful. The guide calls the outputs mostly deterministic, says the APIs are non deterministic by default, and adds a system fingerprint field that changes when OpenAI updates its systems. If that fingerprint changes, the answers may change too. The guide is now marked as archived and refers to older models, so a seed is best seen as a best effort, not a guarantee.
For casual chats, a reworded answer rarely matters. In other settings it matters a lot.
Trust is the first. If a chatbot says yes to a question and then no to the same question a minute later, neither answer deserves much confidence. Fact checking is the second. A claim that holds up across several tries is a better starting point than one that changes each time, though consistency alone does not make something true.
Coding is the third. Developers who use AI to write or fix code want the same request to give the same result, so they can review it, test it and repeat it. Evaluations are the fourth. Researchers and companies compare models by running them on test questions, and the arXiv paper's findings show how much those scores can move from run to run.
There is a useful side to all this. When the same question gets clearly different answers across a few retries, that variation can be a warning sign that the model is unsure. It is the same territory covered in this site's explainer on why AI chatbots still make things up. A wobbly answer is a cue to check a primary source before relying on it.
A few habits make answers far more stable, and none of them require special tools.
Be precise. Vague questions leave room for many reasonable answers. Spell out the task, the context, the length and the format. Google's prompt guide recommends giving clear instructions and constraints, such as asking for a summary in one sentence.
Ask for structure. When the answer needs to follow a pattern, ask for a fixed format like a short list or a table with set columns. For developers, structured output features are the stronger option. Google recommends Gemini's structured output feature for complex JSON, and Anthropic's API offers a JSON schema format.
Rerun to check stability. For anything important, ask the same question two or three times, ideally in fresh chats. If the core answer holds, that is a good sign. If it flips, treat it as uncertain and verify it elsewhere.
Use API settings where they still apply. For older models that support it, a low temperature makes answers more predictable. On OpenAI's API, a fixed seed with identical settings can help, as long as the system fingerprint stays the same. For truly identical output, the work by Thinking Machines Lab suggests it takes changes on the serving side, not just a setting.
Different answers to the same question are not a bug in the usual sense. Some of the variety is designed in on purpose, and some comes from how busy servers do their math. The practical takeaway is simple. Ask clearly, ask for a fixed format, and when an answer matters, ask more than once and treat disagreement as a reason to check.
Comments