Research map / Large language models / Reasoning and chain of thought
Chain-of-Thought Reasoning Without Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pre-trained LLMs contain latent chain-of-thought paths that surface when branching over top-k first tokens, and proposes CoT-decoding, which ranks decoded paths by answer-token confidence margin to pick reasoning paths without any prompt or fine-tuning.
Problem
Reasoning in LLMs is typically elicited via prompting (few-shot or zero-shot CoT) or supervised CoT tuning, which confounds assessment of intrinsic reasoning; the paper asks whether LLMs can reason without prompting.
Approach
Instead of relying on the greedy path, branch over the top-k alternative tokens at the first decoding step under a plain "Q: ... A:" format, then select the path whose final answer has the highest average top-1 vs top-2 probability margin (CoT-decoding), optionally aggregating answers weighted by this margin.
Key results
- On Mistral-7B pre-trained, CoT-decoding raises GSM8K accuracy from 9.9% (greedy) to 25.1%, and to 48.4% when combined with zero-shot CoT prompting.
- On PaLM-2 Large, CoT-decoding reaches 63.2% GSM8K accuracy versus 34.8% greedy, close to the instruction-tuned model's 67.8%.
- Combining CoT-decoding with zero-shot CoT prompting and path aggregation achieves 87.0% on GSM8K for PaLM-2 L, versus 85.3% for self-consistency with zero-shot CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Branch over top-10 first-step tokens and rank paths by answer confidence | Mistral-7B · GSM8K | accuracy: 9.9% -> 25.1% | Helped |
| Use top-k, top-p, beam, or temperature decoding instead of greedy decoding (ablation) | Mistral-7B · GSM8K | accuracy: 9.9% -> 4.9%; 6.4%; 6.7%; 7.5% | Hurt |
| Use confidence-weighted path aggregation instead of selecting the maximum-confidence path (ablation) | CoT-decoding on GSM8K · GSM8K | accuracy: 25.1% -> 25.3% | Helped |
| Add CoT-decoding to greedy decoding | Mistral-7B pretrained · MultiArith | accuracy: 14.3 -> 45.7 | Helped |
| Add CoT-decoding to greedy decoding | Mistral-7B instruction-tuned · Year Parity | accuracy: 62.2 -> 73.5 | Helped |
| Add CoT-decoding to greedy decoding | PaLM-2 Large · Coin Flip (2 rounds) | accuracy: 70.0 -> 94.0 | Helped |
| Add CoT-decoding to greedy decoding | PaLM-2 Large · Sports Understanding | accuracy: 58.8 -> 58.0 | Hurt |
| Add CoT-decoding to greedy decoding | PaLM-2 Large · Object Counting | accuracy: 36.0 -> 39.2 | Helped |
| Add confidence-weighted path aggregation to zero-shot CoT prompting | Mistral-7B · GSM8K | accuracy: 39.4% -> 48.4% | Helped |
| Add confidence-weighted path aggregation to zero-shot CoT prompting | PaLM-2 Large · GSM8K | accuracy: 85.3% -> 87.0% | Helped |
| Increase the number of top alternative first-step tokens considered (ablation) | CoT-decoding · Reasoning benchmarks | accuracy | Helped |
| Increase the number of top alternative first-step tokens considered (ablation) | CoT-decoding on instruction-tuned models · Reasoning benchmarks | accuracy | No clear effect |
Surprising findings
- Greedy decoding often skips the reasoning chain, while CoT paths exist among top-k alternatives at the first token, often ranked low.
- Sampling-based self-consistency without CoT prompts yields little gain, since the model's first token is strongly biased toward direct answers.
Limitations
- Requires exploring k alternative paths, increasing inference cost.
- Answer-confidence margin is defined mainly for short, closed-form answers; less precise for open-ended responses.
- Branching only at the first token; later-step branching is less effective and not fully explored.
Experimental setup
Datasets: GSM8K MultiArith Year Parity (celebrity birth years) Coin Flip Web of Lies Multi-step Arithmetic (BIG-Bench Hard) Sports Understanding Object Counting
Benchmarks: GSM8K MultiArith Year Parity Coin Flip Web of Lies Multi-step Arithmetic Sports Understanding Object Counting
Metrics: accuracy
Baselines: Greedy decoding Top-k sampling Nucleus sampling Beam search Temperature sampling Self-consistency Zero-shot CoT prompting Few-shot CoT prompting
Builds on: chain-of-thought prompting self-consistency zero-shot CoT instruction tuning
Closest papers by meaning
- Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.92)
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.92)
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.91)
- Instance-adaptive Zero-shot Chain-of-Thought Prompting (NeurIPS 2024 · similarity 0.91)
- How Does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding (AAAI 2026 · similarity 0.91)
- Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.