Research map / Large language models / Agents, code and math
OptimalThinkingBench: Evaluating Over and Underthinking in LLMs
Pranjal Aggarwal, Seungone Kim, Jack Lanchantin, Sean Welleck, Jason E Weston, Ilia Kulikov, Swarnadeep Saha
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AdaptThink length-aware RL training | R1-Distill-Qwen-7B · OptimalThinkingBench | F1: 24.5 -> 38.3 | Improved |
| add L1 length-control training | R1-Distill-Qwen-7B · OptimalThinkingBench | F1: 24.5 -> 20.8 | Hurt |
| add SB-DS efficient reasoning | R1-Distill-Qwen-7B · UnderthinkingBench | accuracy: 23.6 -> 14.5 | Hurt |
| add VeriThinker verification training | R1-Distill-Qwen-7B · OptimalThinkingBench | F1: 24.5 -> 27.4 | Mixed |
| merge thinking and non-thinking weights | Qwen3-8B · OptimalThinkingBench | F1: 24.3 -> 38.2 | Improved |
| add L1 length-control training | Qwen3-8B · UnderthinkingBench | accuracy: 47.7 -> 34.6 | Hurt |
| route between thinking and non-thinking with a trained router | Qwen3 family · OptimalThinkingBench | F1: 24.3 -> 46.9 | Improved |
| use oracle router instead of trained router | Qwen3 family · OptimalThinkingBench | F1: 46.9 -> 61.2 | Improved |
| add Don't Overthink prompt suffix | Qwen3 models · OverthinkingBench | thinking tokens: 1493 -> 1147 | Improved |
| add Let's think step-by-step prompt suffix | Qwen3 models · OptimalThinkingBench | F1: 26.3 -> 18.3 | Hurt |
| add irrelevant MCQ distractor options | Qwen3 models · OverthinkingBench | thinking tokens: 42 tokens per option, R^2=0.94 | Hurt |
| enable thinking mode vs non-thinking mode (ablation) | Qwen3-14B · UnderthinkingBench | accuracy: 14% -> 52.4% | Improved |
Closest papers by meaning
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.91)
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Do NOT Think That Much for 2+3=? On the Overthinking of Long Reasoning Models (ICML 2025 · similarity 0.90)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.90)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.90)
- Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
