Research map / Reinforcement learning / RL from feedback
Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Open-Reasoner-Zero, a fully open large-scale RL recipe applying vanilla PPO (GAE λ=γ=1, no KL, binary rule-based reward) directly to Qwen2.5 base models to reproduce R1-Zero-style scaling.
Problem
Reproducing DeepSeek-R1-Zero-style large-scale reasoning RL directly on base models has been closed-source and unstable (e.g., GRPO collapse), limiting community access.
Approach
Use vanilla PPO with GAE(λ=1, γ=1) so the learned critic gives token-level value estimates, drop all KL regularization and format rewards, and scale up curated math/reasoning data; the critic is shown to devalue repetitive patterns, stabilizing training relative to GRPO.
Key results
- ORZ-32B reaches 48.1 on AIME2024, 92.2 on MATH500 and 55.5 on GPQA Diamond, vs 47.0, 91.6, 55.0 for DeepSeek-R1-Zero-Qwen-32B, with about 1/10 the training steps.
- ORZ-32B scores 84.9 on MMLU and 74.4 on MMLU_PRO, above Qwen2.5-32B-Instruct (83.2, 69.2) without instruction tuning.
- ORZ-R1-Distill-Qwen-14B reaches 75.2 on AIME2024 and 95.6 on MATH500, surpassing DeepSeek-R1-Distill-Qwen-32B on AIME2024 (72.6) and MATH500 (94.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Open-Reasoner-Zero training instead of DeepSeek-R1-Zero-Qwen-32B | Qwen2.5-32B-Base · AIME 2024 | accuracy: 47.0 -> 48.1 | Helped |
| Use Open-Reasoner-Zero training instead of DeepSeek-R1-Zero-Qwen-32B | Qwen2.5-32B-Base · MATH500 | accuracy: 91.6 -> 92.2 | Helped |
| Use Open-Reasoner-Zero training instead of DeepSeek-R1-Zero-Qwen-32B | Qwen2.5-32B-Base · GPQA Diamond | accuracy: 55.0 -> 55.5 | Helped |
| Use Open-Reasoner-Zero training instead of Qwen2.5-32B-Instruct | Qwen2.5-32B-Base · MMLU | accuracy: 83.2 -> 84.9 | Helped |
| Use Open-Reasoner-Zero training instead of Qwen2.5-32B-Instruct | Qwen2.5-32B-Base · MMLU_PRO | accuracy: 69.2 -> 74.4 | Helped |
| Add ORZ RL training to DeepSeek-R1-Distill-Qwen-14B | DeepSeek-R1-Distill-Qwen-14B · AIME 2024 | accuracy: 69.7 -> 75.2 | Helped |
| Add ORZ RL training to DeepSeek-R1-Distill-Qwen-14B | DeepSeek-R1-Distill-Qwen-14B · MATH500 | accuracy: 93.9 -> 95.6 | Helped |
| Set GAE lambda to 1.0 rather than 0.95 (ablation) | Open-Reasoner-Zero · training reward | reward | Helped |
| Set GAE lambda to 1.0 rather than 0.95 (ablation) | Open-Reasoner-Zero · response length during training | response length | Helped |
| Remove KL loss and KL penalty (ablation) | ORZ-7B · training performance and response length | training stability | Helped |
| Use the ORZ 57k dataset instead of MATH train 7.5k (ablation) | Open-Reasoner-Zero · training reward | reward | Helped |
| Use the ORZ 57k dataset instead of MATH train 7.5k (ablation) | Open-Reasoner-Zero · response length during training | response length | Helped |
Surprising findings
- Removing KL regularization and format rewards gave stable training; unaligned base models quickly adopt the answer format from a simple binary reward.
- Vanilla GRPO collapsed around 240 steps in the 7B setting with repetitive outputs, while PPO remained stable.
Limitations
- Comparisons to DeepSeek-R1-Zero and DAPO rely partly on reported numbers and differing evaluation metrics.
- Training data reliance on rule-based verification excludes proof-oriented problems.
- Stability analysis of GRPO is based on a single hypothetical GRPO setup and limited ablations.
Experimental setup
Datasets: AIME (up to 2023) MATH Numina-Math Tulu3 MATH OpenR1-Math-220k AoPS forum synthetic general reasoning tasks
Benchmarks: AIME 2024 AIME 2025 MATH500 GPQA Diamond MMLU MMLU_PRO
Metrics: accuracy (averaged over 16 responses)
Baselines: DeepSeek-R1-Zero-Qwen-32B DAPO-Qwen-32B GRPO VAPO DeepSeek-R1-Distill-Qwen-32B Qwen2.5-32B-Instruct
Builds on: PPO GAE GRPO DeepSeek-R1-Zero
Closest papers by meaning
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025 · similarity 0.90)
- AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.90)
- MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model (ICLR 2026 · similarity 0.90)
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data (NeurIPS 2025 · similarity 0.89)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.