Research map / Large language models / Reasoning and chain of thought
Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACPO, an RL framework that adds system-aware fast/slow thinking tokens and an online token-length-budget reward so reasoning models allocate thinking effort by task difficulty and cut overthinking.
Problem
Large reasoning models overthink, producing redundant reasoning regardless of task difficulty, and existing length-control methods either use fixed budgets or offline preference data that ignore difficulty adaptivity.
Approach
Explicit <fast_think>/<slow_think> tags mark thinking modes after SFT cold start; during GRPO training an online token length budget derived from group sampling success rate sets a length reward, and a system pattern reward ties fast/slow token proportions to estimated difficulty.
Key results
- On AIME 2024, ACPO-1.5B reaches 30.0% accuracy with 6670 tokens versus 28.9% and 16894 tokens for DeepSeek-R1-Distill-Qwen-1.5B.
- On MATH 500, ACPO-7B cuts tokens from 3977 to 1405 with accuracy 91.6% vs 92.8% for the base model.
- On MATH 500, ACPO-1.5B scores 81.0% with 1679 tokens versus GRPO's 84.5% with 3098 tokens, so GRPO is more accurate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-1.5B · GSM8K | accuracy: 79.9 -> 81.3 | Helped |
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-7B · GSM8K | accuracy: 86.5 -> 88.3 | Helped |
| Apply ACPO training | DeepSeek-R1-Distill-Llama-8B · GSM8K | accuracy: 82.9 -> 86.7 | Helped |
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-1.5B · MATH 500 | #Token: 5708 -> 1679 | Helped |
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-7B · MATH 500 | accuracy: 92.8 -> 91.6 | Hurt |
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-7B · MATH 500 | #Token: 3977 -> 1405 | Helped |
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-7B · AIME 2024 | ACU: 0.06 -> 0.17 | Helped |
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-1.5B · AIME 2024 | accuracy: 28.9 -> 30.0 | Helped |
| Apply ACPO training | DeepSeek-R1-Distill-Qwen-1.5B · AIME 2024 | #Token: 16894 -> 6670 | Helped |
| Replace the ACPO reward with GRPO reward (ablation) | DeepSeek-R1-Distill-Qwen-1.5B · MATH 500 | #Token: 1679 -> 3098 | Hurt |
| Replace the ACPO reward with GRPO reward (ablation) | DeepSeek-R1-Distill-Qwen-1.5B · MATH 500 | accuracy: 81.0 -> 84.5 | Helped |
| Replace the ACPO reward with GRPO reward (ablation) | DeepSeek-R1-Distill-Qwen-7B · MATH 500 | accuracy: 91.6 -> 92.5 | Helped |
Surprising findings
- GRPO-only training achieves higher MATH 500 accuracy than ACPO for the 1.5B model (84.5% vs 81.0%).
Limitations
- Online difficulty estimation and length budgets rely on verifiable data, limiting generalization to open-domain tasks.
- ACPO introduces accuracy trade-offs compared with the base model on some benchmarks.
- Evaluation limited to math benchmarks and relatively small SFT dataset of 745 samples.
Experimental setup
Datasets: LIMO DeepScaleR-Preview-Dataset GSM8K MATH 500 AIME 2024
Benchmarks: MATH 500 AIME 2024 GSM8K
Metrics: pass@1 accuracy average tokens ACU
Baselines: DeepScaleR-1.5B-Preview THINKPRUNE L1-Exact L1-Max DAST SimPO_DAST GRPO
Builds on: GRPO DAST L1 TOPS DeepScaleR
Closest papers by meaning
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
- SuCo: Sufficiency-guided Continuous Adaptive Reasoning (ICML 2026 · similarity 0.89)
- Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL (NeurIPS 2025 · similarity 0.89)
- Optimizing Anytime Reasoning via Budget Relative Policy Optimization (NeurIPS 2025 · similarity 0.88)
- Think Only When You Need with Large Hybrid-Reasoning Models (NeurIPS 2025 · similarity 0.88)
- DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.