Research map / Large language models / Reasoning and chain of thought
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that mutual-information (HSIC) peaks along LRM reasoning trajectories coincide with thinking tokens ("Wait", "Hmm", "So") that are critical to reasoning, and proposes training-free Representation Recycling and thinking-token test-time scaling to improve math reasoning.
Problem
Reasoning mechanisms of large reasoning models remain a black box; it is unclear which intermediate steps most affect final correctness.
Approach
Track HSIC-estimated mutual information between each generated token's hidden representation and the golden answer's representation; identify sudden MI peaks (IQR outliers), decode them to tokens ("thinking tokens"), and exploit them by recycling their layer representations or appending them to continue reasoning at test time.
Key results
- RR improves DeepSeek-R1-Distill-Llama-8B accuracy by about 20% relatively on AIME24.
- MI peaks occur in at most 5% of reasoning steps (e.g., 0.51% for DeepSeek-R1-Distill-Qwen-7B).
- Suppressing thinking tokens substantially degrades reasoning accuracy while suppressing random non-thinking tokens of equal count has little effect.
- TTTS outperforms the original LRM under equal token budgets on GSM8K and MATH500 and surpasses it on AIME24 beyond 6144 tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reasoning-intensive training (distillation into LRM) vs base model | Llama-3.1-8B / DeepSeek-R1-Distill-Llama-8B · MI trajectories on MATH train | Mean MI: 0.0863 -> 0.1279 | Helped |
| Reasoning-intensive training vs base model | Qwen2.5-14B pair · MI trajectories on MATH train | Mean MI: 1.3128 -> 3.3508 | Helped |
| Reasoning-intensive training vs base model | Qwen2.5-32B pair · MI trajectories on MATH train | Mean MI: 1.7669 -> 4.0352 | Helped |
| Reasoning-intensive training vs base model | Llama-3.3-70B pair · MI trajectories on MATH train | Mean MI: 0.0400 -> 0.0599 | Helped |
| Reasoning-intensive training vs base model (Std and AOM also higher for LRMs) | Llama-3.1-8B pair · MI trajectories on MATH train | AOM: 3.3573 -> 4.5176 | Helped |
| Suppressing thinking tokens (set generation prob to zero) vs randomly suppressing same number of other tokens (ablation) | LRMs on math benchmarks · math reasoning benchmarks | reasoning accuracy | Hurt |
| Representation Recycling (re-run MI-peak representation through same layer once more) | DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B · GSM8K, MATH500, AIME24 | reasoning accuracy | Helped |
| Thinking Token based Test-time Scaling (append thinking token and continue reasoning) | DeepSeek-R1-Distill-Llama-8B · GSM8K, MATH500 under token budgets | reasoning accuracy | Helped |
| Thinking Token based Test-time Scaling | DeepSeek-R1-Distill-Llama-8B · AIME24 under token budgets | reasoning accuracy | Mixed |
Surprising findings
- Random suppression of equal numbers of non-thinking tokens has minimal impact, while suppressing 'Wait'-type tokens can sometimes temporarily improve performance because the model substitutes equivalent phrases like 'But wait'.
Limitations
- Analysis operates at token granularity; semantic-step granularity not explored.
- Origin of MI peaks is not explained mechanistically.
- Theoretical bounds rely on HSIC-based MI estimates with grid-searched kernel bandwidth and are only loosely tied to actual error.
Experimental setup
Datasets: MATH training set (12k problems, 100 sampled for MI trajectories)
Benchmarks: GSM8K MATH500 AIME24
Metrics: accuracy mutual information peak statistics (ratio, interval) Mean/Std/AOM of MI sequences
Baselines: original LRM (DeepSeek-R1-Distill-Llama-8B) DeepSeek-R1-Distill-Qwen-7B original budget-matched original LRM decoding
Builds on: chain-of-thought HSIC critical tokens
Closest papers by meaning
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.92)
- Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens (NeurIPS 2025 · similarity 0.91)
- On Reasoning Strength Planning in Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
- Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning (ICLR 2026 · similarity 0.90)
- PEAR: Phase Entropy Aware Reward for Efficient Reasoning (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.