Research map / Large language models / Reasoning and chain of thought
Thinker: Learning to Think Fast and Slow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Thinker task, a four-stage RL environment (Fast Thinking, Verification, Slow Thinking, Summarization) with per-stage rewards, and trains Qwen2.5-1.5B and DeepSeek-R1-Distill-Qwen-1.5B with PPO on math QA.
Problem
Standard RL on single-turn QA yields long, redundant, imprecise search with weak intuition and verification, because a single sequence-level advantage credits useless search paths alongside useful ones.
Approach
Decompose the QA task within one dialogue into four stages, each with its own binary/weighted reward isolated from the others (discount 0 across stages), so fast intuition, verification, refinement and summarization are trained explicitly; the fast-thinking mode can be used alone under a 1000-token budget.
Key results
- Thinker raises Qwen2.5-1.5B average accuracy from 25.62% (baseline) to 27.33%, with Thinker-Fast reaching 25.18% using under 1000 tokens.
- For DeepSeek-R1-Distill-Qwen-1.5B, Thinker improves average accuracy from 45.90% to 50.98%, an 11.1% relative gain.
- On token-efficiency benchmarks, Thinker-Fast on MATH500 reaches 80.9% with about 600 tokens versus 2780 for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard QA training with the four-stage Thinker task | Qwen2.5-1.5B · Average across eight math benchmarks | Pass@1 accuracy (%): 25.62 -> 27.33 | Helped |
| Replace standard QA training with the four-stage Thinker task | DeepSeek-R1-Distill-Qwen-1.5B · Average across eight math benchmarks | Pass@1 accuracy (%): 45.90 -> 50.98 | Helped |
| Add the Summarization stage to Thinker training (ablation) | Qwen2.5-1.5B Thinker · Average across eight math benchmarks | Pass@1 accuracy (%): 27.89 -> 27.85 | No clear effect |
| Add the Summarization stage to Thinker training (ablation) | Qwen2.5-1.5B Thinker · Fast Thinking | accuracy (%): 24.84 -> 26.75 | Helped |
| Use a 2000-token Verification budget instead of 6000 tokens (ablation) | Thinker · Math benchmarks | performance | No clear effect |
Surprising findings
- Removing the Summarization step lowers Fast accuracy (26.75% to 24.84%) but leaves final accuracy essentially unchanged (27.85% vs 27.84%).
- Thinker's total response length decreases over training for R1.5B as Fast Thinking accuracy rises, instead of growing as in the baseline.
Limitations
- Gains are small for Qwen2.5-1.5B (about 1.7 points average) and some benchmarks (AIME2025, AIME2024 for Q1.5B) are within noise.
- Evaluated only on math-domain QA with 1.5B models in the main experiments; 7B results are only in appendix.
- Verification and summarization rewards rely on heuristic weighting and a tuned coefficient c and length gate.
Experimental setup
Datasets: Open-Reasoner-Zero 129K math QA dataset
Benchmarks: MATH500 AIME2024 AIME2025 GPQA Diamond OlympiadBench AMC23 Minerva Math College Math
Metrics: Pass@1 accuracy averaged over 16 samples standard error average response length reflection pattern count
Baselines: QA task baseline (PPO) Open-Reasoner-Zero SimpleRL ThinkPrune Concise Reasoning SR-FLOW AdaptThink
Builds on: PPO Open-Reasoner-Zero GRPO Thinker MDP
Closest papers by meaning
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.92)
- Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.91)
- QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation (ICLR 2026 · similarity 0.91)
- Thinkless: LLM Learns When to Think (NeurIPS 2025 · similarity 0.91)
- Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.91)
- LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.