Research map / Large language models / Agents, code and math
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WizardMath, trained with RLEIF (Math Evol-Instruct data, instruction reward model and step-level process reward model optimized with PPO) to improve CoT math reasoning without external tools, reaching strong GSM8K and MATH results across model sizes.
Problem
Open-source LLMs lag behind proprietary models on multi-step mathematical reasoning (GSM8K and MATH), with prior SFT-based math data methods susceptible to hallucinated solutions and lacking step-level verification.
Approach
Adapts Evol-Instruct into downward and upward math instruction evolution, uses GPT-4 to label instruction quality (IRM) and step-level correctness (PRM), and trains the policy with step-by-step PPO using the product of instruction and minimum step reward; PRM is also used for verification.
Key results
- WizardMath-Llama 70B reaches 92.8 on GSM8K and 58.6 on MATH, versus MetaMath-70B at 82.3 and 26.6.
- WizardMath-Mistral 7B reaches 90.7 on GSM8K and 55.4 on MATH, versus MetaMath-Mistral 7B at 77.9 and 28.6.
- Adding PRM and then IRM in PPO improves Mistral-7B SFT from 82.8 to 90.7 on GSM8K and from 48.1 to 55.4 on MATH.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RLEIF to Llama-2 70B rather than use the strongest listed same-size baseline | Llama-2 70B · GSM8K and MATH | pass@1: 92.8 vs. 90.6; 58.6 vs. 48.6 | Helped |
| Apply RLEIF to Mistral-v0.1 7B rather than MetaMath | Mistral-v0.1 7B · GSM8K and MATH | pass@1: 90.7 vs 77.9; 55.4 vs 28.6 | Helped |
| Add PRM to SFT during PPO training (ablation) | WizardMath-SFT · GSM8K and MATH | pass@1: 77.4 -> 81.7; 35.6 -> 39.9 | Helped |
| Add IRM to PRM-based PPO training (ablation) | WizardMath-SFT + PRM · GSM8K and MATH | pass@1: 77.4 -> 84.1; 35.6 -> 43.5 | Helped |
| Replace PRM with ORM during PPO training (ablation) | WizardMath-SFT · GSM8K and MATH | pass@1: 81.7 vs 79.1; 39.9 vs 36.8 | Hurt |
| Replace PRM with PRM800k during PPO training (ablation) | WizardMath-SFT · GSM8K and MATH | pass@1: 81.7 vs 79.7; 39.9 vs 38.7 | Hurt |
| Replace PRM with Math-Shepherd during PPO training (ablation) | WizardMath-SFT · GSM8K and MATH | pass@1: 81.7 vs 80.3; 39.9 vs 38.2 | Hurt |
| Use Llama3 rather than GPT-4 to generate Math Evol-Instruct data (ablation) | WizardMath-SFT, Mistral-7B-v0.1 · GSM8K and MATH | pass@1: 82.8 -> 76.7; 48.1 -> 43.5 | Hurt |
| Add two rounds of downward evolution to original training data (ablation) | Mistral-7B SFT · GSM8K and MATH | pass@1: 59.7 vs. 74.5; 15.1 vs. 34.7 | Helped |
| Add three rounds of upward evolution to original training data (ablation) | Mistral-7B SFT · GSM8K and MATH | pass@1: 59.7 vs. 78.6; 15.1 vs. 42.5 | Helped |
| Combine downward evolution with three rounds of upward evolution (ablation) | Mistral-7B SFT · GSM8K and MATH | pass@1: 78.6 vs. 81.2; 42.5 vs. 46.2 | Helped |
| Use PRM rather than Self-Consistency for verification (ablation) | SFT + PRM generator · GSM8K and MATH, pass@256 | accuracy: 92.3 -> 95.2; 59.3 -> 64.7 | Helped |
Surprising findings
- Open-source Llama-3-70B-Instruct used in place of GPT-4 for evolution yields a 5-6 point gap, but still gives large gains over the base model.
- Self-consistency verification with majority voting underperforms the PRM verifier, with the gap widening on MATH at larger candidate counts.
Limitations
- Reward signals rely on GPT-4 labels, so the pipeline depends on a proprietary model and may inherit its errors.
- Reported table includes several models and results at mixed settings, making direct comparisons across rows hard to verify.
- Math-specific gains on MATH remain well below proprietary top models such as GPT-4o on some subsets.
Experimental setup
Datasets: GSM8K training set MATH training set MwpBench
Benchmarks: GSM8K MATH MwpBench
Metrics: pass@1 (greedy CoT) pass@256
Baselines: MetaMath MAmmoTH MathScale Xwin-Math Math-Shepherd PRM800k GPT-3.5-Turbo GPT-4 Claude 2 Gemini Pro Skywork-Math DART-Math
Builds on: Evol-Instruct WizardLM WizardCoder Let's Verify Step by Step (PRM800K) Math-Shepherd
Closest papers by meaning
- MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy (AAAI 2026 · similarity 0.90)
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking (ICLR 2026 · similarity 0.89)
- DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning (ICLR 2026 · similarity 0.88)
- We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.88)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.88)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.