Research map / Large language models / Reasoning and chain of thought
AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that large-scale RL (math-only then code-only, GRPO with strict on-policy updates and stage-wise length extension) substantially improves distilled DeepSeek-R1-Distill-Qwen 7B/14B models on math and code reasoning.
Problem
Open reproductions of large-scale RL for reasoning lack consistent recipes, and prior work suggests distillation beats RL for small and mid-sized models.
Approach
Start RL from strong distilled SFT models, run math-only RL first (which also improves code), then code-only RL, using strict on-policy GRPO (REINFORCE-style, no KL), stage-wise response length extension 8K->16K->24K->32K, and difficulty-based prompt filtering.
Key results
- AceReason-Nemotron-14B reaches 78.6% on AIME24 and 67.4% on AIME25, up from 69.7%/50.2% for its distilled starting point.
- Math-only RL on the 7B model raises LiveCodeBench v5 from 37.6% to 44.4%, and full training reaches 51.8%.
- AceReason-Nemotron-14B scores 61.1% on LiveCodeBench v5, above OpenCodeReasoning-14B and DeepCoder-14B (57.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply math-only RL (8K→24K) | DeepSeek-R1-Distill-Qwen-7B · AIME 2025 | accuracy: 39.0 -> 52.5 | Helped |
| apply math-only RL (8K→24K) | DeepSeek-R1-Distill-Qwen-7B · LiveCodeBench v5 | accuracy: 37.6 -> 44.4 | Helped |
| apply math-only RL (8K→24K) | DeepSeek-R1-Distill-Qwen-14B · AIME 2025 | accuracy: 50.2 -> 63.4 | Helped |
| apply math-only RL (8K→24K) | DeepSeek-R1-Distill-Qwen-14B · LiveCodeBench v5 | accuracy: 53.1 -> 58.6 | Helped |
| continue with code-only RL (24K→32K) | 7B after math-only RL · AIME 2025 | accuracy: 52.5 -> 51.7 | Hurt |
| continue with code-only RL (24K→32K) | 7B after math-only RL · LiveCodeBench v5 | accuracy: 44.4 -> 51.8 | Helped |
| continue with code-only RL (24K→32K) | 14B after math-only RL · AIME 2024 | accuracy: 76.6 -> 75.7 | Hurt |
| continue with code-only RL (24K→32K) | 14B after math-only RL · LiveCodeBench v5 | accuracy: 58.6 -> 61.1 | Helped |
| train on Hard prompts instead of the full prompt set (ablation) | AceReason-Nemotron-7B at 24K stage · AIME 2024 | accuracy: 63.3 -> 65.9 | Helped |
| train on Hard prompts instead of the full prompt set (ablation) | AceReason-Nemotron-7B at 24K stage · AIME 2025 | accuracy: 51.1 -> 52.5 | Helped |
| start at 8K, then extend to 16K and 24K rather than starting at 24K (ablation) | AceReason-Nemotron-7B math RL · AIME 2024 | accuracy | Helped |
| start directly at 24K rather than progressively extending response length (ablation) | AceReason-Nemotron-7B math RL · AIME 2024 | accuracy | Hurt |
Surprising findings
- Math-only RL improves code reasoning on LiveCodeBench, not just math benchmarks.
- RL improves pass@k up to k=1024 over the distilled SFT model, contrary to the common view that RL only improves pass@1.
Limitations
- Experiments limited to 7B and 14B models starting from DeepSeek-R1-Distill-Qwen checkpoints.
- Math-only RL shows a small AIME25 drop on the 7B model after code RL (-0.8), and code RL cuts AIME24 for 14B (-0.9).
- Code RL is sensitive to false positive and false negative test-case rewards.
Experimental setup
Datasets: DeepScaleR NuminaMath LiveCodeBench AtCoder/LeetCode/Aizu competitive programming problems
Benchmarks: AIME2024 AIME2025 MATH500 HMMT2025 BRUMO2025 LiveCodeBench v5 LiveCodeBench v6 LiveCodeBench Pro (Codeforces ELO) EvalPlus
Metrics: avg@64 pass@1 avg@8 pass@1 pass@k Codeforces ELO and percentile
Baselines: DeepSeek-R1-Distill-Qwen-7B DeepSeek-R1-Distill-Qwen-14B OpenMath-Nemotron-14B OpenCodeReasoning-Nemotron-14B DeepCoder-14B Skywork-OR1-Math-7B Light-R1-14B Llama-Nemotron-Ultra-253B
Builds on: GRPO DeepSeek-R1 DeepScaleR veRL LiveCodeBench
Closest papers by meaning
- AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.94)
- MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model (ICLR 2026 · similarity 0.93)
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
- Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.92)
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.