Research map / Large language models / Reasoning and chain of thought

Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InfoBias and InfoGain information-theoretic metrics for reasoning efficiency in LRMs and an entropy-based Adaptive Think strategy that halts reasoning once answer-distribution entropy is low, cutting tokens by about 50-68% on math with small accuracy gains.

Problem

Large reasoning models generate excessively long chains of thought, and it is unclear when extended reasoning stops adding value; the paper seeks to quantify reasoning efficiency and shorten reasoning without hurting accuracy.

Approach

Two information-theoretic metrics: InfoBias (mutual information between generated and ideal reasoning trajectories, estimated by HSIC) at the response level and InfoGain (entropy reduction over candidate answers per reasoning step) at the step level. Based on these, an Adaptive Think strategy stops reasoning when average answer-distribution entropy falls below a threshold alpha/(e ln2).

Key results

• On QwQ-32B, Adaptive Think reduces tokens by 66.46% on GSM8K and 68.25% on AIME2025 versus Vanilla Think, with accuracy gains of +0.75% and +0.93%.
• Across four additional benchmarks, QwQ-32B Adaptive Think (alpha=0.1) averages +1.23% accuracy and 42.52% fewer tokens than Vanilla Think (paper text), though MMLU-Pro and MuSR results at alpha=0.1 show accuracy drops for DeepSeek-R1-32B.
• On CommonsenseQA, DeepSeek-R1-32B Adaptive Think reduces tokens by 80.99% relative to Vanilla Think.

Measured effects reported in the paper

Surprising findings

• No-Think mode yields higher final confidence than Vanilla Think on CommonsenseQA, suggesting intermediate reasoning can be redundant.
• Models show intuitive confidence in the correct answer before any reasoning begins (step 0).

Limitations

• Requires access to next-token probability distributions, so closed models such as o1 can only be analyzed via sampling approximations.
• Not applicable to open-ended questions lacking a single correct answer.
• Accuracy drops on some knowledge-intensive tasks (e.g., MMLU-Pro, ProntoQA) at aggressive thresholds.

Experimental setup

Datasets: GSM8K AIME2025 MMLU-Pro MuSR ProntoQA CommonsenseQA

Benchmarks: GSM8K AIME2025 MMLU-Pro MuSR ProntoQA CommonsenseQA

Metrics: accuracy average token count InfoBias InfoGain

Baselines: Vanilla Think No-Think Gated Think CoT prompting Base prompting

Builds on: chain-of-thought prompting vLLM HSIC

Closest papers by meaning

• Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities (AAAI 2026 · similarity 0.92)
• Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.91)
• Mitigating Overthinking in Large Reasoning Language Models via Reasoning Path Deviation Monitoring (NeurIPS 2026 · similarity 0.91)
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.91)
• A*-Thought: Efficient Reasoning via Bidirectional Compression for Low-Resource Settings (NeurIPS 2025 · similarity 0.90)