Research map / Large language models / Reasoning and chain of thought
Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests whether tasks where verbal deliberation hurts humans also show CoT-induced performance drops in LLMs/LMMs, across six psychology-inspired task archetypes.
Problem
Under which settings chain-of-thought systematically reduces performance in LLMs and LMMs remains unclear; the paper seeks a heuristic for predicting such cases.
Approach
Use six tasks from cognitive psychology where verbal thinking or deliberation impairs human performance as a heuristic for finding CoT failures; scale up each task and evaluate zero-shot vs CoT across frontier models, finding large drops in three archetypes (implicit statistical learning, visual face recognition, learning rules with exceptions) and mixed effects in the other three.
Key results
- On artificial grammar learning, GPT-4o zero-shot 87.5% vs CoT 64.4%, and o1-preview subset reached 57.7% vs GPT-4o zero-shot 94.0% on the same subset (36.3-point drop).
- On facial recognition, CoT reduced accuracy for all six LMMs tested, e.g. GPT-4o 64.00% to 51.20%.
- On the vehicle exception-learning task, CoT increased rounds for GPT-4o from 2.9 to 12.5 (331% more).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoT instead of zero-shot prompting | GPT-4o · Artificial grammar learning | accuracy: 87.50% -> 64.40%; 23.10% | Hurt |
| Use built-in inference-time reasoning instead of GPT-4o zero-shot | o1-preview · Artificial grammar learning (subset) | accuracy: 94.00% -> 57.70%; 36.30% | Hurt |
| Use CoT instead of zero-shot prompting | Claude 3.5 Sonnet · Artificial grammar learning | accuracy: 65.90% -> 67.70%; -1.80% | No clear effect |
| Use CoT instead of zero-shot prompting | GPT-4o · Facial recognition | accuracy: 64.00% -> 51.20%; 12.80% | Hurt |
| Use CoT instead of zero-shot prompting | Claude 3.5 Sonnet · Facial recognition | accuracy: 97.80% -> 94.80%; 3.00% | Hurt |
| Use CoT before each prediction instead of direct prompting | GPT-4o · Vehicle classification with exceptions | average rounds to learn labels: 2.9 -> 12.5; 9.6 | Hurt |
| Use CoT instead of zero-shot prompting | GPT-4o · Logical inconsistency (MNLI) | accuracy: 53.2% -> 93.9% | Helped |
| Use CoT instead of zero-shot prompting | GPT-4o · Logical inconsistency (SNLI) | accuracy: 51.4% -> 94.3% | Helped |
| Use CoT instead of zero-shot prompting | GPT-4o · Spatial intuitions | accuracy: 38% -> 40%; +2% | No clear effect |
| Use CoT instead of zero-shot prompting | GPT-4o · Apartment selection (Δ [0.1, 0.3]) | accuracy: 47% -> 45% | No clear effect |
| Use CoT instead of zero-shot prompting | Claude 3.5 Sonnet · Apartment selection (Δ [0.5, 1]) | accuracy: 81% -> 95% | Helped |
| Use tree-of-thought instead of zero-shot prompting (ablation) | GPT-4o · Artificial grammar learning (subset) | accuracy: 94.00% -> 64.55% | Hurt |
Surprising findings
- o1-preview with built-in reasoning underperformed zero-shot GPT-4o on implicit statistical learning by 36.3 points.
- CoT improved performance on logical inconsistency and apartment selection tasks, where human deliberation hurts performance.
Limitations
- Heuristic from human psychology does not perfectly predict model behavior; human-model differences matter.
- Task implementations were scaled and modified, so failures may depend on implementation choices.
- o1-preview evaluated only on small subsets due to cost; tested only a limited set of prompts.
Experimental setup
Datasets: SNLI MNLI
Benchmarks: human overthinking benchmark
Metrics: accuracy rounds to learn labels bootstrap p-value
Baselines: zero-shot prompting tree-of-thought
Builds on: chain-of-thought prompting tree-of-thought
Closest papers by meaning
- Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond (ICLR 2026 · similarity 0.89)
- How Does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding (AAAI 2026 · similarity 0.88)
- Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.88)
- Test-time Prompt Intervention (AAAI 2026 · similarity 0.87)
- Thought Branches: Interpreting LLM Reasoning Requires Resampling (ICLR 2026 · similarity 0.87)
- Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.