Research map / Large language models / Reasoning and chain of thought
DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents DAPO, an open-source large-scale RL system and algorithm (decoupled clip, dynamic sampling, token-level loss, overlong reward shaping) that trains Qwen2.5-32B to 50 AIME 2024 points with half the steps of DeepSeek-R1-Zero-Qwen-32B.
Problem
Large-scale RL for long chain-of-thought LLM reasoning is hard to reproduce: naive GRPO on Qwen2.5-32B reaches only about 30 AIME points, suffering entropy collapse, reward noise, and training instability, while key recipe details of reasoning models are undisclosed.
Approach
A GRPO-style policy-gradient algorithm with four modifications: asymmetric clipping ranges (Clip-Higher) to prevent entropy collapse, dynamic sampling that filters zero-advantage groups, token-level loss normalization for long sequences, and overlong reward shaping (overlong filtering and soft length penalty) to reduce reward noise; all code, data and recipe are released.
Key results
- DAPO reaches 50 avg@32 on AIME 2024 with Qwen2.5-32B, versus 47 for DeepSeek-R1-Zero-Qwen-32B, using 50% of its training steps.
- Naive GRPO reaches 30 on AIME 2024; adding overlong filtering gives 36, Clip-Higher 38, soft overlong punishment 41, token-level loss 42, and dynamic sampling 50.
- Clip-Higher raises the upper clip to 0.28 (lower 0.2) to counter entropy collapse and keep entropy from dropping.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Overlong Filtering (ablation) | Naive GRPO on Qwen2.5-32B · AIME24 avg@32 | accuracy: 30 -> 36 | Helped |
| Add Clip-Higher (ablation) | Naive GRPO with Overlong Filtering on Qwen2.5-32B · AIME24 avg@32 | accuracy: 36 -> 38 | Helped |
| Add Soft Overlong Punishment (ablation) | Naive GRPO with Overlong Filtering and Clip-Higher on Qwen2.5-32B · AIME24 avg@32 | accuracy: 38 -> 41 | Helped |
| Add Token-level Loss (ablation) | Naive GRPO with Overlong Filtering, Clip-Higher, and Soft Overlong Punishment on Qwen2.5-32B · AIME24 avg@32 | accuracy: 41 -> 42 | Helped |
| Add Dynamic Sampling (ablation) | Qwen2.5-32B with Overlong Filtering, Clip-Higher, Soft Overlong Punishment, and Token-level Loss · AIME24 avg@32 | accuracy: 42 -> 50 | Helped |
| Use DAPO instead of DeepSeek-R1-Zero-Qwen-32B | Qwen2.5-32B · AIME24 avg@32 | accuracy: 47 -> 50 | Helped |
Surprising findings
- Token-level loss gives less accuracy gain than other techniques but mainly improves training stability and healthier length growth.
- Dynamic sampling increases sampling cost but overall training time is not significantly affected, since generation is dominated by long-tail samples.
Limitations
- null
Experimental setup
Datasets: DAPO-Math-17K
Benchmarks: AIME 2024
Metrics: avg@32 accuracy
Baselines: Naive GRPO DeepSeek-R1-Zero-Qwen-32B
Builds on: PPO GRPO verl DeepSeek-R1
Closest papers by meaning
- DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization (NeurIPS 2025 · similarity 0.91)
- Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.91)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.91)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model (NeurIPS 2025 · similarity 0.90)
- Nudging the Boundaries of LLM Reasoning (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.