Research map / Large language models / Reasoning and chain of thought

Modeling Hierarchical Thinking in Large Reasoning Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Models LRM chain-of-thought as a six-state finite state machine, derives a correct-vs-incorrect transition advantage matrix, and uses Q-value-guided orthogonal activation steering at sentence boundaries to improve reasoning accuracy with fewer interventions.

Problem

Reasoning trajectories of large reasoning models are poorly understood and can be inconsistent or pathological; there is no global map of how cognitive modes transition and no principled way to decide when and where to intervene to improve correctness.

Approach

Abstract CoT sentences into six cognitive states forming an FSM; estimate outcome-conditioned transition advantage matrix R; treat steering as a planning problem solved by Q-value iteration over states; apply training-free orthogonal activation steering with transition vectors (difference-of-means) at sentence boundaries, gated by Q-gap and classifier confidence.

Key results

• On AIME25, GPT-L accuracy rises from 43.33% to 56.67% with Q-value steering using 55.20 average interventions versus 83.87 for weighted steering.
• On MATH-500, GPT-L reaches 83.20% with Q-value steering using 0.48 interventions per sample versus 12.17 for greedy steering (about 25x fewer).
• On GPQA Diamond, GPT-M improves from 64.14% to 67.17% with Q-value steering.

Surprising findings

• Second-order (two-step history) transition modeling gave little or no accuracy benefit over the first-order Markov design, supporting the memoryless FSM choice.
• Greedy steering sometimes reduced accuracy relative to the default (e.g., QWEN on AIME25 from 83.33% to 76.67%).

Limitations

• Memoryless six-state FSM is a coarse, fixed abstraction that may not transfer to other domains; state labels depend on GPT-4o-mini annotation and classifier errors.
• Correlational advantage matrix does not establish that steering into positive-advantage states causes correct answers.
• Steering is limited to sentence boundaries detected by punctuation and requires layer selection on a hold-out set.

Experimental setup

Datasets: AIME25 GPQA Diamond MATH-500 GSM8K

Benchmarks: AIME25 MATH-500 GSM8K GPQA Diamond

Metrics: accuracy average tokens average steering interventions

Baselines: greedy steering weighted steering default (no steering) prompt-based transition guidance

Builds on: activation steering (difference-of-means) Q-value iteration Polya four-step framework Schoenfeld episode theory

Closest papers by meaning

• Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation (ICML 2025 · similarity 0.90)
• Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
• State of Thought Enables Endogenous Reasoning (NeurIPS 2026 · similarity 0.90)
• Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning (ICLR 2026 · similarity 0.90)
• TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning (NeurIPS 2026 · similarity 0.90)
• Dynamic Early Exit in Reasoning Models (ICLR 2026 · similarity 0.90)