Research map / Large language models / Agents, code and math
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Absolute Zero, a zero-data RLVR paradigm where one model proposes and solves code reasoning tasks (induction, abduction, deduction) verified by a code executor, and instantiates it as AZR trained on Qwen2.5 and Llama3.1 models.
Problem
Existing zero-setting RLVR methods still depend on human-curated question-answer collections, which raises scalability concerns and may limit learning for future superhuman AI systems.
Approach
A single LLM acts as both proposer and solver in a self-play loop; a code executor validates self-proposed Python triplets (program, input, output) and provides verifiable rewards, with a learnability reward for the proposer and a binary correctness reward for the solver, trained with Task-Relative REINFORCE++.
Key results
- AZR-Coder-7B reaches 50.4 overall average and 61.6 coding average, the best 7B average among compared zero-style models, exceeding the previous best by 1.8 points overall.
- AZR-Base-7B improves math average by 10.9 points over its Qwen2.5-7B base (38.4 vs 27.5) without any math training data.
- Scaling the coder models gives +5.7, +10.2, and +13.2 overall average gains for 3B, 7B, and 14B respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with self-proposed tasks and no curated data | Qwen2.5-7B-Coder · OOD coding and math benchmarks | Overall average: 40.2 -> 50.4 | Helped |
| Train with self-proposed tasks and no curated data | Qwen2.5-7B · OOD coding and math benchmarks | Overall average: 39.8 -> 46.8 | Helped |
| Train on deduction tasks only, removing abduction and induction (ablation) | AZR · OOD math benchmarks | Math average: 38.4 -> 32.0 | Hurt |
| Remove induction tasks (ablation) | AZR · OOD math benchmarks | Math average: 38.4 -> 33.3 | Hurt |
| Remove dynamic conditioning on K historical reference triplets and use a fixed prompt (ablation) | AZR proposer · OOD coding benchmarks | Code average: 55.2 -> 54.4 | Hurt |
| Remove dynamic conditioning on K historical reference triplets and use a fixed prompt (ablation) | AZR proposer · OOD math benchmarks | Math average: 38.4 -> 33.1 | Hurt |
| Do not train the proposer; train only the solver (ablation) | AZR · OOD coding and math benchmarks | Overall average: 46.8 -> 45.4 | Hurt |
| Add the error-deduction task (ablation) | AZR · Downstream performance | Performance | No clear effect |
| Use composite functions as curriculum learning (ablation) | AZR · Downstream performance | Performance | No clear effect |
| Initialize the seed buffer from the LeetCode Dataset instead of self-generated data (ablation) | AZR · Mathematical reasoning | Performance | Hurt |
| Add complexity rewards (ablation) | AZR proposer · Downstream performance | Performance | No clear effect |
| Add code-edit-distance or surprise-based diversity rewards (ablation) | AZR · Downstream performance | Performance | No clear effect |
Surprising findings
- Coder-initialized AZR surpasses the base-initialized variant on math after training even though it starts lower on math (23.9 vs 27.5).
- Llama3.1-8B trained with AZR occasionally produces safety-concerning chains of thought, termed the 'uh-oh moment'.
Limitations
- Safety concerns: Llama3.1-8B produced unsafe chains of thought and the paper does not address safe management of self-improving systems.
- Only deterministic programs are considered; code execution safety relies on a forbidden-module filter rather than sandboxing.
- Gains on Llama3.1-8B are more limited, and the approach is evaluated mainly on Qwen2.5 models.
Experimental setup
Datasets: self-generated code triplets (program, input, output) CruxEval-I CruxEval-O LiveCodeBench-Execution
Benchmarks: HumanEval+ MBPP+ LiveCodeBench v1-5 AIME'24 AIME'25 AMC'23 MATH500 Minerva OlympiadBench CruxEval-I CruxEval-O LiveCodeBench-Execution
Metrics: pass@1 (greedy) pass@k average accuracy
Baselines: Qwen2.5-7B-Instruct Qwen2.5-7B-Coder Qwen2.5-Math-7B AceCoder CodeR1 PRIME-Zero SimpleRL-Zoo Oat-Zero ORZ
Builds on: REINFORCE++ GRPO DeepSeek-R1-Zero SimpleRL-Zoo veRL
Closest papers by meaning
- Learning to Reason without External Rewards (ICLR 2026 · similarity 0.91)
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
- Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.90)
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model (NeurIPS 2025 · similarity 0.89)
- Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.