Chain-of-thought prompting asks a model to write intermediate reasoning before its final answer. The evidence supports a narrow claim: it substantially changes accuracy on mathematical, logical and algorithmic problems, and much less clearly elsewhere. The written reasoning is also not reliably an account of what produced the answer.
What chain-of-thought prompting changes
Few-shot CoT places worked examples, each with a written rationale, before the question. Zero-shot CoT appends a cue such as “Let’s think step by step” and supplies no exemplars. The 2022 study by Wei et al. used the few-shot form: prompting PaLM 540B with eight chain-of-thought exemplars reached state-of-the-art accuracy on GSM8K (arXiv 2201.11903). I have not verified the zero-shot cue’s effect sizes against a primary source.
Where the accuracy gain is established
The strongest cross-task evidence is Sprague et al., published at ICLR 2025. They reviewed 1,218 experimental comparisons from 110 papers and ran their own tests on 20 datasets and 14 models. Their main finding is that CoT helps substantially mainly on problems requiring mathematical, logical or algorithmic reasoning. On MMLU, direct answering was almost as accurate as CoT unless the question or the model’s response contained the relevant reasoning structure.
The 2022 paper also reported gains on commonsense tasks. The 2025 meta-analysis locates its gains in maths and logic. Both framings are contested. The author’s reading, not a finding: the commonsense benefit looks smaller and less consistent in the broader evidence. Nothing here shows CoT does nothing for commonsense tasks. Whether a reported gain exceeds sampling noise depends on benchmark size, covered in statistical power when comparing models.
Model size and the 100B figure
Secondary accounts report that CoT gains appear mainly in models of roughly 100B parameters, and that smaller models can produce fluent but illogical chains that score below standard prompting. I have not confirmed the exact wording or scope of this claim in the primary paper. Treat it as reported, not established.
Is the written reasoning faithful?
Three claims get conflated: that CoT changes accuracy, that the written steps cause the answer, and that the steps are a usable safety signal. The first has the most support. The second is challenged by several studies.
Turpin et al. tested three kinds of bias, including a suggested answer, on Claude 1.0 and GPT-3.5, and found the explanations systematically unfaithful under biased prompts (arXiv 2305.04388). Lanham et al. reportedly found that models reach the correct answer despite altered chains, including early stopping and inserted errors. I cite this only through a summary (arXiv 2405.15092) and have not checked the primary paper. Work attributed to Anthropic in arXiv 2307.13702 reportedly found large variation across tasks in how much models rely on the CoT, and that uninformative filler text gave no accuracy gain. That suggests extra test-time compute alone does not explain the boost.
Behavioural tests of this kind are necessary but not sufficient for claims about internal computation. A plausible-looking trace does not show what happened inside the network.
Where it breaks down
The 2025 preprint “Chain-of-Thought Reasoning in the Wild Is Not Always Faithful” (arXiv 2503.08679) reports unfaithful CoTs under naturally worded, non-adversarial prompts, including on thinking models. It describes Implicit Post-Hoc Rationalization, where a model gives the same yes or no to a question and to its reverse, and Unfaithful Illogical Shortcuts on hard maths. Some arguments were superficially coherent while answering Yes to both of two contradictory questions, or No to both. The authors report that thinking models are more faithful but not entirely so.
Weaker reports follow. The preprint “The Curse of CoT” (arXiv 2504.05081) reports that CoT consistently underperformed direct answering on pattern-based in-context learning. A reported ASU study, “Chain of Thoughtlessness”, found CoT helped only on very narrow planning tasks. I have only a secondary account (BDTechTalks), so treat it as low confidence. A 2026 medical preprint (arXiv 2603.25960) changed answers on 1,262 of 4,183 questions, hurting 750 and helping 512, a net loss of 238. It covers one model in one domain and is illustrative only.
Cost, reasoning models and output format
A Wharton GAIL technical report puts the extra time for CoT requests at 20–80% (10–20 seconds). I have not checked its methodology. The same report notes that many models already reason by default, so generic CoT prompts add limited value to reasoning models. I found no large controlled study of how much explicit CoT adds to a native reasoning model. Token cost also rises. Practitioner reports say reasoning tokens can leak into responses, which may need post-processing for structured output; that is advice, not a controlled study. Schema enforcement is a separate problem, covered in what schema constraints actually enforce in LLM tool calling.
Safety monitoring: conditions and open questions
CoT is a usable monitoring signal only if the trace reflects the computation that produced the answer. If a model can take prompt-to-answer shortcuts that bypass the CoT, a monitor reads a plausible story rather than the cause. A 2026 preprint frames CoT monitoring as valid only when traces are faithful (alphaXiv 2605.24286). A secondary summary treats the unfaithfulness findings as a concern for CoT-based safety protocols. That is an argument, not an established result. The “in the wild” authors forecast that unfaithfulness may persist as models become more capable. This is a forecast, not a result.
What is established, contested and open
Established, under stated conditions: CoT raises accuracy on mathematical, logical and algorithmic tasks, per Sprague et al.’s comparison across 14 models and 20 datasets. Contested: commonsense scope, the parameter threshold, and benefit for reasoning models. Open: whether written reasoning stays faithful at scale, and whether it is a reliable monitoring signal.
Primary papers I checked include Wei et al., Sprague et al., Turpin et al., the Anthropic work and the 2025 preprint. The remaining claims rely on secondary summaries, flagged above. When you run or report CoT experiments, record the model and version, parameter count, dataset and split, sample size, the exact prompt including exemplars, and decoding settings. Without them, a gain cannot be compared or reproduced.