Research map / Large language models / Reasoning and chain of thought
Chain of Thoughtlessness? An Analysis of CoT in Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Case study showing chain-of-thought prompts help LLMs on Blocksworld planning only when prompts are highly problem-specific, and gains collapse as stack size grows beyond demonstrated examples; similar failures appear in scaled CoinFlip, letter concatenation, and arithmetic tasks.
Problem
Whether chain-of-thought prompting teaches LLMs general algorithmic procedures that transfer out of distribution to larger or more general planning and reasoning instances.
Approach
Systematically varies CoT prompt specificity (zero-shot, progression proof, Blocksworld universal algorithm, stacking, lexicographic stacking) and problem size on scalable Blocksworld instances, and builds scalable variants of CoinFlip, LastLetterConcatenation, and single-digit arithmetic to test whether CoT gains generalize in length.
Key results
- On table-to-stack Blocksworld, GPT-4 zero-shot achieves 3.8% while the stacking CoT prompt reaches 59.3% on 261 instances, but only the most specific prompt retains gains.
- Blocksworld CoT gains do not generalize beyond a handful of blocks; accuracy drops sharply as goal stack size grows regardless of prompt specificity.
- On scaled CoinFlip, GPT-4-Turbo with CoT stays near-perfect up to 4 steps and dips below 90% only around 31-step problems, while letter concatenation and arithmetic show performance collapsing with length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the table-to-stack-specific Stacking Prompt instead of zero-shot prompting | GPT-4-Turbo · table-to-stack Blocksworld (261 instances) | accuracy: 19.1% -> 40.6% | Helped |
| Use the table-to-stack-specific Stacking Prompt instead of zero-shot prompting | GPT-4 · table-to-stack Blocksworld (261 instances) | accuracy: 3.83% -> 59.3% | Helped |
| Use a general progression-proof CoT prompt instead of Domain-Specific n-shot examples | GPT-4-Turbo · table-to-stack Blocksworld (261 instances) | accuracy: 13.7% -> 15.3% | Mixed |
| Use the Blocksworld Universal Algorithm instead of Domain-Specific n-shot examples | GPT-4 · table-to-stack Blocksworld (261 instances) | accuracy: 6.13% -> 51.3% | Helped |
| Add zero-shot CoT | GPT-4-Turbo · CoinFlip | accuracy: 56.38% -> 95.71% | Helped |
| Add manual CoT examples | GPT-4-Turbo · LastLetterConcatenation | accuracy: 10.00% -> 51.06% | Helped |
| Add manual CoT examples | GPT-4-Turbo · single-digit arithmetic | accuracy: 24.13% -> 50.43% | Helped |
| Add manual CoT examples | GPT-4-Turbo · LastVowelConcatenation | accuracy: 5.75% -> 27.00% | Helped |
| Add manual CoT examples | GPT-4-Turbo · FoomLetterConcatenation | accuracy: 1.81% -> 26.00% | Helped |
| Add manual CoT examples | GPT-4-Turbo · arithmetic with single-digit intermediate answers | accuracy: 45.60% -> 69.31% | Helped |
| Give explicit single-digit intermediate-answer information with zero-shot CoT | GPT-4-Turbo · arithmetic with single-digit intermediate answers | accuracy: 45.60% -> 42.76% | Hurt |
| Use incorrect rather than manual CoT examples (ablation) | GPT-4-Turbo · CoinFlip | accuracy: 98.89% -> 96.76% | Hurt |
Surprising findings
- Self-consistency performed similar to or worse than standard CoT on table-to-stack planning problems.
- Incorrect CoT prompts in CoinFlip performed nearly as well as correct manual CoT (96.76% vs 98.89%), suggesting content matters little.
Limitations
- Only a few closed API models were tested, and no error bars were reported due to cost.
- Synthetic domains are very simple, limiting how far conclusions extend to realistic reasoning.
- Code and data were not released in the paper's checklist, though a GitHub link is given in the abstract.
Experimental setup
Datasets: Blocksworld (PDDL-generated instances) CoinFlip LastLetterConcatenation GSM8K
Benchmarks: Blocksworld table-to-stack CoinFlip (scaled) LastLetterConcatenation (scaled) LastVowelConcatenation FoomLetterConcatenation single-digit multi-step arithmetic
Metrics: accuracy Levenshtein distance
Baselines: zero-shot prompting zero-shot CoT n-shot prompting self-consistency Fast Downward
Builds on: chain-of-thought prompting zero-shot CoT self-consistency PDDL
Closest papers by meaning
- Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.91)
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.91)
- On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.90)
- Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning (NeurIPS 2025 · similarity 0.90)
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.90)
- Understanding Chain-of-Thought in LLMs through Information Theory (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.