Research map / Large language models / Reasoning and chain of thought
Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IBPO, a constrained RL method that caps the share of extended (costly) reasoning responses under an inference budget, implemented as a weighted SFT update where weights come from a per-batch integer linear program over CGPO-style feasible sets, and evaluated on MATH500 with LLaMA 3.1 8B.
Problem
Long reasoning models spend unnecessarily long reasoning on trivial math questions, raising inference cost; the paper seeks to make reasoning length adapt to query difficulty under an explicit inference budget.
Approach
Formulate response-group density caps (standard CoT vs. extended responses) as a constrained policy optimization problem; solve a per-batch margin-maximization integer program in the non-parametric space and project it onto the LLM via a weighted SFT (semi-gradient) update, which lets the model allocate costly responses to harder prompts.
Key results
- ASV-IuB with q+=50% and 75% improves pass@1 over SFT baselines while reducing cost by 4.14% at 2.16x trials and 5.74% at 4.32x trials, on par with SCoRe.
- Budget constraints are followed on the training set almost exactly; on the test set, voting ratio orders 75% > 50% > 25%.
- At q+=25%, only about 2% of level-1 MATH problems receive voting responses, showing difficulty-aware allocation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use early-stopping sequential voting instead of vanilla majority voting, measuring performance by number of responses | LLaMA 3.1 8B on MATH500 · MATH500 | majority@k: nearly 10% | Helped |
| Measure SV performance accounting for token cost rather than only number of responses | SV versus vanilla MV · MATH500 | performance-cost efficiency | No clear effect |
| Optimize ASV with IBPO under a 50% extended-response budget | ASV-SFT-1 on MATH500 · MATH500 | performance-cost efficiency: 4.14% at 2.16× | Helped |
| Optimize ASV with IBPO under a 75% extended-response budget | ASV-SFT-1 on MATH500 · MATH500 | performance-cost efficiency: 5.74% at 4.32× | Helped |
| Set q_+ to 75% rather than 25% or 50% (ablation) | ASV-IuB on MATH500, last checkpoint · MATH500 | performance-budget efficiency | No clear effect |
| Set the extended-response budget to 25% or 50% (ablation) | ASV-IuB on MATH500 · MATH500 by difficulty level | voting-response allocation: 2% of level 1 problems | Helped |
| Replace IBPO's budget-aware optimization with vanilla CGPO | LLaMA on MATH500 · MATH500 | pass@1 | Hurt |
Surprising findings
- CGPO with plain LLaMA on the SDPO prompt set gives little gain, while the budget-aware IBPO variant achieves noticeable gains.
- SFT-based and prompting-based adaptive methods fall below the efficiency boundary, suggesting SFT alone does not learn resource allocation.
Limitations
- Evaluated only on MATH500 with one 8B model and a small 10k-prompt RL set; test-set distribution shift weakens budget adherence at q+=75%.
- Limited to two response groups and one RL algorithm (CGPO); no reward models used.
Experimental setup
Datasets: MATH SDPO (step-DPO dataset) MATH500
Benchmarks: MATH500
Metrics: pass@1 majority@k average number of trials voting ratio vs. difficulty level
Baselines: Self-Refine STaR S3C SFT-RI SFT-SCoRe SCoRe SV-SFT ASV-SFT-1 LLaMA 3.1 8B Instruct majority voting
Builds on: CGPO RAFT RFT self-consistency Step-DPO
Closest papers by meaning
- Optimizing Anytime Reasoning via Budget Relative Policy Optimization (NeurIPS 2025 · similarity 0.90)
- In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback (AAAI 2026 · similarity 0.90)
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
- Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.90)
- Thinkless: LLM Learns When to Think (NeurIPS 2025 · similarity 0.90)
- SuCo: Sufficiency-guided Continuous Adaptive Reasoning (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.