Research map / Large language models / Reasoning and chain of thought
What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes ten large reasoning models' CoT traces on HARP and GPQA-Diamond, finding that within-question shorter length, lower review ratio, and especially lower failed-step fraction (from a Claude-extracted reasoning graph) correlate with accuracy, and that FSF-based reranking and failed-branch removal causally improve accuracy.
Problem
Prior work disagrees on whether longer chain-of-thought traces and more review behaviors (induced by appended 'wait' tokens) improve reasoning accuracy in large reasoning models. The paper asks what structural properties of CoTs characterize effective reasoning.
Approach
Controls for question-level confounders via within-question conditional correlations and GLMMs, extracts reasoning graphs by prompting Claude 3.7 to emit Graphviz with failed nodes marked, and proposes the Failed-Step Fraction as the strongest predictor, validated via test-time reranking and CoT editing that removes failed branches.
Key results
- Within the same question, shorter CoTs, lower Review Ratio, and lower Failed-Step Fraction correlate with higher accuracy; FSF is significant for all ten models on both datasets.
- FSF-based top-1 selection among 64 samples yields roughly 5-13% gains over random on AIME 2025.
- Removing failed branches from incorrect CoTs raises continuation accuracy by roughly 8-14% (e.g., DeepSeek R1 first failed branch 20.89% to 29.42%).
Surprising findings
- Claude 3.7 shows a positive association between Review Ratio and accuracy on math, the opposite of other models.
- Length-based selection on GPT oss 120B does not improve accuracy, and Qwen 3 shows anomalous behavior on HARP that is attributed to likely training contamination.
Limitations
- Correlational analysis is test-time only; training effects on FSF are not studied.
- Reasoning graphs and failed-branch labels come from a single LLM (Claude 3.7) extraction; faithfulness of CoTs is not assessed.
- Some results (e.g., Qwen on HARP) are suspected to be affected by train-evaluation contamination.
Experimental setup
Datasets: HARP GPQA-Diamond AIME 2025
Benchmarks: HARP GPQA-Diamond AIME 2025
Metrics: accuracy pass@1 conditional correlation GLMM coefficient
Baselines: Random selection Length-based selection Review Ratio-based selection
Builds on: S1 test-time scaling with wait tokens GLMM
Closest papers by meaning
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.91)
- Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.91)
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.90)
- TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning (NeurIPS 2026 · similarity 0.90)
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.