Research map / Large language models / Reasoning and chain of thought
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta-analyzes 110 CoT papers and runs zero/few-shot CoT vs direct-answer experiments on 20 datasets and 14 LLMs, finding CoT helps mainly on math and symbolic reasoning and that tool-augmented solvers beat CoT on those tasks.
Problem
Prompt-based chain-of-thought is widely assumed to help on nearly all reasoning tasks, but it is unclear for which task types CoT actually improves over direct answering and why.
Approach
Combines a literature meta-analysis of CoT-vs-direct comparisons with controlled experiments across many datasets and models, then splits CoT's gain into planning and execution stages and compares against tool-augmented solvers. It finds the gain concentrates in symbolic execution (e.g., MMLU questions containing '='), and that symbolic solvers outperform CoT.
Key results
- CoT gains concentrate on math and symbolic tasks: average improvement of 14.2 points for symbolic, 12.3 for math, and 6.9 for logical reasoning in the literature meta-analysis, versus near-zero for other categories (56.8 vs 56.1).
- On MMLU, about 95% of CoT's total gain is attributed to questions containing '=' in the question or generated output.
- Plan + Tool Solver (PAL/SatLM) substantially outperforms CoT and Plan + CoT Solver on math and logic datasets across most models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use zero-shot CoT instead of direct answering | 14-model average · Mathematical reasoning | accuracy: 22.6 -> 50.2 | Helped |
| Use zero-shot CoT instead of direct answering | 14-model average · Symbolic reasoning | accuracy: 45.2 -> 56.1 | Helped |
| Use zero-shot CoT instead of direct answering | 14-model average · Commonsense reasoning | accuracy: 74.8 -> 75.7 | No clear effect |
| Use zero-shot CoT instead of direct answering | 14-model average · Knowledge | accuracy: 73.3 -> 77.5 | Mixed |
| Use zero-shot CoT instead of direct answering | 14-model average · Soft reasoning | accuracy: 58.3 -> 60.3 | Mixed |
| Use CoT instead of direct answering | Tasks involving symbolic, math, or logical reasoning in the literature · Symbolic, math, and logical reasoning | performance: 45.5 -> 56.9 | Helped |
| Use CoT instead of direct answering | Other task categories in the literature · Other task categories | performance: 56.1 -> 56.8 | No clear effect |
| Use few-shot CoT instead of zero-shot CoT (ablation) | Reasoning datasets · Reasoning datasets | performance | No clear effect |
Surprising findings
- On commonsense, reading comprehension, and knowledge datasets (e.g., CSQA, PIQA, SIQA, WinoGrande, ARC), zero-shot CoT yields little to no gain over direct answering.
- Few-shot CoT does not qualitatively change which datasets benefit from CoT compared with zero-shot.
Limitations
- Does not cover long-horizon planning tasks, except BiGGen Bench.
- Possible data contamination of models on the evaluated datasets is not fully addressed.
- Meta-analysis relies on manually extracted comparisons from 110 papers and is subject to selection and categorization choices.
Experimental setup
Datasets: CommonsenseQA StrategyQA SIQA PIQA Winogrande GPQA MuSR ContextHub ARC AGIEval LSAT MMLU MMLU Pro
Benchmarks: CommonsenseQA StrategyQA SIQA PIQA Winogrande GPQA MuSR ContextHub ARC AGIEval LSAT MMLU MMLU Pro
Metrics: accuracy CoT delta over direct answer paired bootstrap significance
Baselines: direct answer prompting Plan + Direct Solver Plan + CoT Solver Plan + Tool Solver PAL SatLM
Builds on: chain-of-thought prompting Program-aided Language Models Satisfiability-Aided Language Models
Closest papers by meaning
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.92)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.92)
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.92)
- Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.92)
- Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.92)
- Chain-of-Thought Reasoning Without Prompting (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.