Research map / Large language models / Reasoning and chain of thought
How Far Are We from Optimal Reasoning Efficiency?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines empirical reasoning efficiency frontiers and a Reasoning Efficiency Gap (REG) metric from many fine-tuned LRMs, then proposes REO-RL, an RL method that optimizes length-constrained rewards over a small set of exponentially spaced token budgets to reduce REG.
Problem
Large reasoning models produce verbose chain-of-thought traces, and existing efficiency fine-tuning methods are hard to compare because their evaluations are inconsistent; it is unclear how close they are to the optimal length-accuracy trade-off.
Approach
Builds empirical efficiency frontiers as the per-budget maximum accuracy over many fine-tuned models, defines REG as the area gap between a model's budget-accuracy curve and that frontier, and trains with a trapezoidal-rule approximation of the summed length-constrained reward over a few token budgets (exponentially spaced in REO-RL (Exp)).
Key results
- REO-RL (Exp) reduces REG by at least 50% across all evaluated LRMs relative to the base or vanilla RL models (e.g., 57.6% average REG reduction on DeepSeek-R1-Distill-Qwen-1.5B).
- On DeepSeek-R1-Distill-Qwen-7B, REO-RL (Exp) keeps average accuracy at 62.9% (vs 64.4% vanilla RL) while cutting average length from 11282.6 to 5966.0 tokens.
- REO-RL (Exp) matches Qwen3-4B/8B efficiency frontiers under a 16K budget with average accuracy drop of 2.8 and 2.5 points respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply REO-RL (Exp) with exponentially spaced token budgets | DeepSeek-R1-Distill-Qwen-1.5B · Average across AIME 2024, AIME 2025, AMC 2023, and Minerva Math | REG reduction: 57.6% | Helped |
| apply REO-RL (Exp) with exponentially spaced token budgets | DeepSeek-R1-Distill-Qwen-7B · Average across AIME 2024, AIME 2025, AMC 2023, and Minerva Math | REG reduction: 55.9% | Helped |
| apply REO-RL (Exp) with exponentially spaced token budgets | Qwen3-4B · Average across AIME 2024, AIME 2025, AMC 2023, and Minerva Math | REG reduction: 65.2% | Helped |
| apply REO-RL (Exp) with exponentially spaced token budgets | Qwen3-8B · Average across AIME 2024, AIME 2025, AMC 2023, and Minerva Math | REG reduction: 75.9% | Helped |
| use oracle greedy token-budget selection instead of exponential spacing (ablation) | REO-RL on DeepSeek-R1-Distill-Qwen-7B · Average | REG reduction: 55.9% -> 58.0% | Helped |
| use question-specific minimum token budgets (ablation) | REO-RL on DeepSeek-R1-Distill-Qwen-7B · Average | REG reduction: 55.9% -> 59.3% | Helped |
| use question-specific hard minimum token budgets (ablation) | REO-RL on DeepSeek-R1-Distill-Qwen-7B · Average | REG reduction: 55.9% -> 65.6% | Helped |
| use linearly spaced rather than exponentially spaced token budgets (ablation) | REO-RL on DeepSeek-R1-Distill-Qwen-7B · Average | REG reduction: 55.9% -> 50.3% | Hurt |
| increase selected token budgets from the default to K=10 (ablation) | REO-RL on DeepSeek-R1-Distill-Qwen-7B · Average | REG reduction: 55.9% -> 48.1% | Hurt |
| set all objective coefficients to 1 (ablation) | REO-RL on DeepSeek-R1-Distill-Qwen-7B · Average | accuracy (%): 64.4 -> 62.1 | Hurt |
| set all objective coefficients to 1 (ablation) | REO-RL on DeepSeek-R1-Distill-Qwen-7B · Average | REG reduction: 55.9% -> 69.3% | Helped |
| use RL with length-harmonizing rewards | Qwen3-4B · Average | REG reduction: 12.3% | Hurt |
Surprising findings
- RL with length-harmonizing rewards enlarges the efficiency gap for Qwen3-4B/8B models.
- Hybrid reasoning (HGPO) does not necessarily reduce length; the model reverts to long thinking outside think tags.
Limitations
- Frontiers are empirical lower bounds that depend on the set of fine-tuned models, so REG is relative to that set.
- Fine-tuning LRMs to precisely match the efficiency frontiers remains unsolved; Qwen3 models still show accuracy loss.
- Evaluation limited to mathematical benchmarks (AMC 2023, AIME 2024/2025, Minerva Math).
Experimental setup
Datasets: DeepScaleR AReaL
Benchmarks: AIME 2024 AIME 2025 AMC 2023 Minerva Math
Metrics: accuracy average response length reasoning efficiency gap (REG)
Baselines: RL with token budgets length group normalized reward length-harmonizing reward MRT HGPO SFT shortest SFT TOPS SimPO DAST BRPO
Builds on: PPO SimPO AReaL s1 budget forcing
Closest papers by meaning
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.92)
- Promoting Efficient Reasoning with Verifiable Stepwise Reward (AAAI 2026 · similarity 0.91)
- Efficient Reasoning with Balanced Thinking (ICLR 2026 · similarity 0.91)
- Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty (ICLR 2026 · similarity 0.91)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.91)
- On Reasoning Strength Planning in Large Reasoning Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.