Research map / Large language models / Agents, code and math
OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds OpenMathInstruct-2, a 14M-pair commercially permissive math SFT dataset synthesized with Llama3.1-405B-Instruct, with ablations on CoT format, teacher choice, noise, and question diversity; fine-tuned Llama3.1 8B/70B base models reach 67.8% and 71.9% on MATH.
Problem
Math reasoning SFT data is largely closed-source or generated with restrictive-license GPT models, and the design choices (solution format, teacher model, filtering, question diversity) that drive SFT performance are poorly understood.
Approach
Use an open-weight frontier model (Llama3.1-405B-Instruct) to both augment solutions for MATH/GSM8K questions and synthesize new questions, guided by controlled ablations (OpenMath CoT format, coverage matching, noise injection) and LLM-based paraphrase decontamination.
Key results
- OpenMath2-Llama3.1-8B reaches 67.8% on MATH, outperforming Llama3.1-8B-Instruct (51.8%) by 15.9 points; 76.1% with maj@256.
- OpenMath2-Llama3.1-70B reaches 71.9% on MATH, 3.9 points above Llama3.1-70B-Instruct (67.9%).
- Teacher 405B-Instruct data yields 37.9% vs 30.1% for 8B-Base teacher data at matched coverage; OpenMath CoT format gets 44.5% vs 40.6% for Llama CoT with 40% shorter solutions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenMath CoT rather than Llama CoT (ablation) | SFT data for Llama3.1-8B-Base · MATH validation | accuracy: 40.6 ± 0.6 -> 44.5 ± 0.8 | Helped |
| Use OpenMath CoT rather than Llama CoT (ablation) | SFT data for Llama3.1-8B-Base · MATH validation | mean solution length (tokens): 331.3 -> 237.0 | Helped |
| Generate SFT solutions with Llama3.1-405B-Instruct rather than Llama3.1-8B-Base (ablation) | Llama3.1-8B-Base SFT · MATH validation | accuracy: 30.1 ± 0.6 -> 37.9 ± 0.6 | Helped |
| Filter low-quality solutions with LLM-as-a-Judge Prompt 1 (ablation) | Llama3.1-8B-Base SFT · MATH validation | accuracy: 43.6 ± 1.7 -> 43.6 ± 0.1 | No clear effect |
| Filter low-quality solutions with LLM-as-a-Judge Prompt 2 (ablation) | Llama3.1-8B-Base SFT · MATH validation | accuracy: 43.6 ± 1.7 -> 43.0 ± 0.8 | No clear effect |
| Filter solutions using Nemotron helpfulness ≥ 3 (ablation) | Llama3.1-8B-Base SFT · MATH validation | accuracy: 43.6 ± 1.7 -> 43.8 ± 0.4 | No clear effect |
| Filter solutions using Nemotron correctness ≥ 3 (ablation) | Llama3.1-8B-Base SFT · MATH validation | accuracy: 43.6 ± 1.7 -> 43.1 ± 0.4 | No clear effect |
| Include up to 20% incorrect solutions during SFT (ablation) | Llama3.1-8B-Base SFT · MATH validation | accuracy | No clear effect |
| Increase unique questions from 1K to 6.5K while holding SFT pair count fixed (ablation) | Llama3.1-8B-Base SFT · MATH validation | accuracy: +10.5% | Helped |
| Set the minimum majority-vote threshold to 8 rather than 0 (ablation) | Question-solution augmented SFT data · MATH validation | accuracy: 50.1 -> 49.2 | Hurt |
| Set the minimum majority-vote threshold to 16 rather than 0 (ablation) | Question-solution augmented SFT data · MATH validation | accuracy: 50.1 -> 44.4 | Hurt |
| Set the minimum majority-vote threshold to 24 rather than 0 (ablation) | Question-solution augmented SFT data · MATH validation | accuracy: 50.1 -> 42.0 | Hurt |
Surprising findings
- SFT performance stayed robust with up to 20% incorrect solutions added, and LLM/reward-model filtering of low-quality solutions gave no meaningful gain.
- Few-shot prompting of an aligned Llama-Instruct model reverted to its trained Llama CoT format for 57% of generations, fixed by dropping the role special tokens (base template).
Limitations
- 70B model only improves on a subset of benchmarks; design choices were tuned on the 8B model.
- 70B trained only on 5M subset due to computational constraints.
- Automatic filtering of low-quality solutions is imperfect, making the effect of removal hard to gauge.
Experimental setup
Datasets: MATH training set GSM8K training set AMC 2023 AIME 2024 Omni-MATH synthesized questions from MATH and GSM8K
Benchmarks: GSM8K MATH AMC 2023 AIME 2024 Omni-MATH
Metrics: accuracy (greedy) maj@256 accuracy
Baselines: Llama3.1-8B-Instruct Llama3.1-70B-Instruct NuminaMath-7B-CoT NuminaMath-72B-CoT Qwen2.5-Math-7B-Instruct Qwen2.5-Math-72B-Instruct Mathstral-7B DS-Coder-V2-Lite-Instruct
Builds on: NuminaMath OpenMathInstruct-1 MetaMathQA Llama 3.1 Nemotron-4-340B-Reward
Closest papers by meaning
- OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset (NeurIPS 2024 · similarity 0.94)
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning (ICML 2024 · similarity 0.91)
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (ICLR 2024 · similarity 0.91)
- JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.91)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.91)
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.