Research map / Large language models / Reasoning and chain of thought
Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoThink, which uses an ellipsis prompt to stochastically toggle R1-style models between thinking and no-thinking modes, then applies three-stage GRPO with batch balancing, accuracy-focused, and length-aware rewards to learn adaptive reasoning.
Problem
R1-style large reasoning models overthink, spending many tokens even on simple problems; prompt-only or uniform pruning methods either fix a mode manually or degrade accuracy on hard problems.
Approach
An ellipsis in the think tag ("<think>\n...\n") stochastically triggers thinking or no-thinking; a three-stage RL schedule then stabilizes the dual-mode coexistence with batch-level reward balancing, reinforces accuracy, and prunes length with length-aware rewards so the model learns difficulty-aware thinking.
Key results
- On DeepSeek-R1-Distill-Qwen-1.5B, AutoThink-Stage3 reaches 51.7% average accuracy versus 48.6% for the standard prompt, with 5108 tokens versus 10633 (about 52% fewer).
- On DeepScaleR-Preview-1.5B, AutoThink-Stage2 reaches 57.3% average accuracy versus 56.7% for the standard prompt while using about 10% fewer tokens (5277 vs 5817).
- On Skywork-OR1-Math-7B, AutoThink-Stage3 cuts average tokens from 9053 to 3974 with accuracy 68.8% versus 70.5% for the standard prompt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Stage 2 after Stage 1, before pruning (ablation) | AutoThink on Distill-R1-1.5B · Five math benchmarks | accuracy (%): 47.6% -> 51.7% | Helped |
| Add length-aware reward shaping in Stage 3 (ablation) | AutoThink on Distill-R1-1.5B · Five math benchmarks | Token Usage: 7295 -> 5108 | Helped |
| Add Stage 2 accuracy reinforcement before Stage 3 pruning (ablation) | AutoThink on DeepScaleR-Preview-1.5B · Five math benchmarks | accuracy (%): 56.7 -> 57.3 | Helped |
| Add Stage 3 length-aware pruning after Stage 2 (ablation) | AutoThink on DeepScaleR-Preview-1.5B · Five math benchmarks | accuracy (%): 57.3 -> 55.7 | Hurt |
| Use ellipsis prompt instead of standard prompt | DeepSeek-R1-Distill-Qwen-1.5B · Five math benchmarks | accuracy (%): 48.6 -> 44.2 | Hurt |
| Use ellipsis prompt instead of standard prompt | DeepSeek-R1-Distill-Qwen-1.5B · Five math benchmarks | Token Usage: 10633 -> 7453 | Helped |
| Use no-thinking prompt instead of standard prompt | DeepSeek-R1-Distill-Qwen-1.5B · Five math benchmarks | accuracy (%): 48.6 -> 37.5 | Hurt |
| Add batch-level reward balancing in Stage 1 (ablation) | AutoThink · Stage 1 training | mode stability | Helped |
| Replace naive Stage 1 reward with naive reward plus length-aware brevity shaping (ablation) | AutoThink · Stage 1 training | thinking/no-thinking mode stability | Hurt |
| Use naive reward without batch balancing (ablation) | AutoThink · Stage 1 training | thinking/no-thinking mode stability | Hurt |
| Add the Think By Difficulty instruction to the prompt (ablation) | Distill-R1 models · Five math benchmarks | accuracy | Hurt |
| Vary Stage 1 target balance ratio gamma from 0.5 to 0.8 (ablation) | AutoThink · Stage 1 training, step 200 | Thinking-Rate (%): 54.2 -> 100.0 | Helped |
Surprising findings
- Simply inserting an ellipsis into the think tag stochastically triggers no-thinking mode without any training, yet the no-thinking rate does not track problem difficulty.
- Skipping Stage 2 and pruning directly after Stage 1 leaves accuracy stagnant at 47.6% and causes response length to rebound.
Limitations
- Reasoning-related tokens still appear after the </think> tag in no-thinking mode, indicating reward hacking and incomplete mode separation.
- Response length cannot be controlled by an explicit budget.
- Training data is not filtered by difficulty.
Experimental setup
Datasets: DeepScaleR dataset (40K math problems)
Benchmarks: MATH500 Minerva OlympiadBench AIME24 AMC23 GPQA MMLU LiveCodeBench
Metrics: pass@1 accuracy average token usage Efficiency-F1 perplexity
Baselines: Standard prompt No-thinking prompt Ellipsis prompt Concise-RL ShorterBetter ThinkPrune
Builds on: GRPO DeepScaleR GRPO-LEAD DeepSeek-R1
Closest papers by meaning
- R-4B: Incentivizing General-Purpose Auto-Thinking in MLLMs via Bi-Mode Annealing and Reinforce Learning (CVPR 2026 · similarity 0.91)
- RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.90)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
- Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.89)
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.89)
- Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.