Image credit: OpenAI, Learning to reason with LLMs, September 2024. Official figure, resized to 16:9.
Chatbots now often include a thinking toggle. Flip it on and the model spends longer before it answers. Flip it off and it replies faster. That switch is not a marketing gimmick. It reflects a real shift in how AI systems spend computing power.
For years, the main way to improve language models was to train bigger ones on more data. Researchers call that training time scaling. A newer approach adds a second axis. The model can also spend more compute at the moment it answers. Researchers call that test time compute. The idea is simple. Give the model room to write out steps, try several attempts, or check its own work, and accuracy often rises. The catch is that answers get slower and costlier, and recent work shows the model can overthink easy questions.
Classic scaling laws tracked loss against the amount of training compute. Bigger models and longer pretraining runs bought most of the gains users noticed. OpenAI's September 2024 post on learning to reason with LLMs makes the newer split explicit. The company says the performance of its o1 model consistently improves with more reinforcement learning at train time and with more time spent thinking at test time. Those two levers are not the same. Training changes the weights once. Test time compute is spent again for every query.
That difference matters for readers. A larger model costs more to train and to run. A thinking mode keeps the same model and burns extra tokens only when you ask. Snell and colleagues at UC Berkeley and Google DeepMind argued in 2024 that scaling test time compute optimally can be more effective than scaling model parameters in some settings. On problems where a smaller base model already has non trivial success rates, they report that test time compute can outperform a model about 14 times larger in a FLOPs matched comparison. Gains depend on how hard the prompt is. Easy and intermediate questions often favor extra thinking. The hardest ones still benefit more from a stronger pretrained model.
One early way to spend answer time compute is to ask the model to show its work. Wei and colleagues at Google Research showed in 2022 that chain of thought prompting, giving a few examples that include intermediate reasoning steps, unlocks much stronger performance on arithmetic, commonsense, and symbolic tasks. The gains emerge mainly in large models. With PaLM 540B and just eight chain of thought exemplars, they report state of the art accuracy on the GSM8K math word problem benchmark, surpassing even a finetuned GPT 3 system that used a verifier. An ablation in that paper is telling. Prompting the model to emit only a string of dots matched to equation length, a crude way to force extra tokens without natural language steps, did about as well as standard prompting. Variable computation alone was not enough. The useful part was expressing intermediate steps in language.
Wang and colleagues then showed that sampling many chains of thought and taking a majority vote over final answers helps further. Their self consistency method replaces greedy decoding with sample and marginalize. On PaLM 540B they report absolute gains such as plus 17.9 percent on GSM8K, plus 11.0 percent on SVAMP, plus 12.2 percent on AQuA, plus 6.4 percent on StrategyQA, and plus 3.9 percent on ARC challenge. The method needs no extra training or human labels. It simply spends more inference samples and keeps the most common answer. That is parallel test time compute.
Later work by Brown and colleagues under the Large Language Monkeys title pushes the same idea across huge sample budgets. They find that coverage, the fraction of problems solved by any generated sample, often scales with the number of samples over four orders of magnitude. On SWE bench Lite, DeepSeek Coder V2 Instruct rises from 15.9 percent of issues solved with one sample to 56 percent with 250 samples, above a reported single sample state of the art of 43 percent. Coverage can keep climbing even when practical ways of picking the right sample plateau. On MATH with Llama 3 8B Instruct, coverage rises from 82.9 percent at 100 samples to 98.44 percent at 10,000 samples, while majority voting or reward model selection only moves from about 40.50 percent to 41.41 percent over that same range.
OpenAI's o1 post describes a model trained with large scale reinforcement learning so that it learns to use a long internal chain of thought. The company reports that o1 ranks in the 89th percentile on Codeforces competitive programming questions, places among the top 500 students nationally on a USA Math Olympiad qualifier, and exceeds human PhD level accuracy on GPQA diamond. On the 2024 AIME exams, GPT 4o solved about 12 percent of problems on average, while o1 averaged 74 percent with a single sample, 83 percent with consensus among 64 samples, and 93 percent when re ranking 1000 samples with a learned scoring function. The post also states that o1 was the first model on that GPQA diamond setup to surpass recruited PhD experts, with the caveat that this does not mean o1 is more capable than a PhD in every respect. OpenAI also decided not to show the raw chain of thought to users and instead shows a model generated summary.
DeepSeek AI's R1 paper shows a related path with open weights. DeepSeek R1 Zero is trained with pure reinforcement learning from a base model without a supervised fine tuning stage first. Rewards come from rule based checks of final answer correctness and format. On AIME 2024, the authors report that pass at 1 rises from 15.6 percent to 71.0 percent during training, and that majority voting further improves the score to 86.7 percent. Average response length on the training set grows as the model learns to spend more thinking time. They describe an aha moment when the model begins to use the word wait during reflections. The paper also notes that DeepSeek R1 is sensitive to prompts and that few shot prompting consistently degrades its performance, so the authors recommend clear zero shot instructions.
Not every extra token is equally useful. Muennighoff and colleagues introduce budget forcing in the s1 paper. After supervised fine tuning Qwen2.5 32B Instruct on only 1000 curated reasoning examples, they control thinking length at decode time. To cut thinking short they append an end of thinking delimiter. To lengthen it they suppress that delimiter and append Wait so the model continues. With that control, s1 32B exceeds o1 preview on some competition math settings by up to 27 percent on MATH and AIME24 in their reporting, and scaling with budget forcing raises AIME24 from 50 percent to 57 percent beyond the no intervention baseline.
Snell and colleagues make a complementary point. The best way to spend a fixed inference budget depends on prompt difficulty. On easier problems, sequential revision of an initial answer can beat sampling many independent answers. On harder problems, parallel search or tree search against a process reward model can be better. Their compute optimal policy improves efficiency by more than four times compared with a best of N baseline in their experiments. Predicted difficulty bins that do not use ground truth labels still capture much of the gain. The practical lesson is that a single always on thinking mode is a blunt tool. Matching strategy to difficulty matters.
Chen and colleagues study overthinking in o1 like models directly. They find that models such as QwQ 32B Preview and DeepSeek R1 Preview often allocate many tokens and many solution rounds to simple questions with little benefit. For the question what is the answer of 2 plus 3, they report that o1 like models consumed about 1953 percent more tokens than conventional models to reach the same answer, and they show a QwQ response with 13 solutions. Across mathematical benchmarks, later solutions often contribute little to accuracy. In cases where the model eventually gets the answer right, the first correct answer appears in the first solution round more than 92 percent of the time. Distinctness of reasoning strategies also falls in later rounds. Their self training approach that prefers shorter, first correct solutions with limited reflection cuts token output by 48.6 percent on MATH500 for QwQ while maintaining accuracy in their reported setting.
Brown and colleagues add another caveat that matters for products. Without automatic verifiers, majority voting and reward model selection often plateau even while oracle coverage keeps rising. Generating many candidates is only useful if you can identify the good ones. That gap is why coding tasks with unit tests convert extra samples into real wins more cleanly than open ended word problems.
Thinking modes spend tokens. OpenAI's o1 post notes that the model produces a long internal chain of thought before responding and that the company shows users a summary rather than the raw trace. DeepSeek R1 Zero generates longer responses as training progresses. Self consistency and repeated sampling multiply that cost by the number of samples. Snell's FLOPs matched framing and Brown's cost comparisons both treat inference tokens as a scarce budget. That is why paid plans with weekly meters feel tighter when thinking is on. Coverage we have published on Grok Bot usage draining faster after a meter rollback and on Elon Musk saying Grok Bot cost too much to serve sits in the same economic pattern. Extra answer time compute is useful, and it is also what the meter is metering.
A separate question is whether the visible thinking is faithful. Our earlier guide Can you trust an AI's thinking asks whether reasoning traces show what the model actually used. This piece is about the compute tradeoff instead. Even when traces are incomplete or summarized, spending more test time compute can still raise accuracy on hard problems, as OpenAI and DeepSeek report. Readers should not confuse a longer looking explanation with a free lunch.
Use a thinking or high effort mode when the task is multi step and easy to check. Math contests, coding with tests, logic puzzles, and technical debugging are the settings where Wei, Wang, OpenAI, DeepSeek, Snell, Brown, and Muennighoff all show gains from extra answer time compute. Leave it off for short factual lookups, formatting, or simple arithmetic where Chen's overthinking results say later rounds rarely change the answer. If a product exposes an effort slider, treat medium as the default and reserve maximum for problems you would yourself solve with scratch paper. If you can run several samples and automatically verify them, coverage scaling from Brown and colleagues is often a better buy than one very long chain. If you cannot verify, stop after a modest number of samples because selection methods plateau.
The research consensus is not that longer thinking always wins. It is that answer time compute is a real scaling axis with diminishing and sometimes negative returns on easy work. Training still builds the knowledge. Thinking spends that knowledge more carefully when the problem is hard enough to need it.
Comments