Research map / Large language models / Agents, code and math
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Bootstraps GSM8K and MATH questions via LLM rephrasing, backward reasoning (SV, FOBAR) and answer augmentation into MetaMathQA, then fine-tunes LLaMA-2 into MetaMath models that set new open-source results on GSM8K and MATH.
Problem
Open-source LLMs such as LLaMA-2 perform poorly on mathematical reasoning (GSM8K, MATH), and existing finetuning data is limited in question diversity and lacks backward reasoning questions.
Approach
Augment the math training set along two axes: rephrase questions and create backward-reasoning questions (masking a number as x, via Self-Verification and FOBAR) while also sampling multiple correct answer paths, then finetune LLaMA-2 on the merged MetaMathQA dataset.
Key results
- MetaMath-7B reaches 66.5% on GSM8K and 19.8% on MATH, versus 54.9% and 10.7% for prior 7B WizardMath.
- MetaMath-70B (QLoRA) reaches 82.3% on GSM8K, slightly above GPT-3.5-Turbo's 80.8%.
- Adding SV and FOBAR backward questions to AnsAug and rephrasing raises LLaMA-2-7B GSM8K accuracy from 60.6% to 64.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune on MetaMathQA instead of the original base model | LLaMA-2-7B · GSM8K | accuracy: 14.6 -> 66.5 | Helped |
| Finetune on MetaMathQA instead of the original base model | LLaMA-2-7B · MATH | accuracy: 2.5 -> 19.8 | Helped |
| Add rephrasing to answer augmentation (ablation) | LLaMA-2-7B trained on GSM8K · GSM8K | accuracy: 59.6 -> 60.6 | Helped |
| Add self-verification and FOBAR to answer augmentation and rephrasing (ablation) | LLaMA-2-7B trained on GSM8K · GSM8K | accuracy: 60.6 -> 64.4 | Helped |
| Add self-verification and FOBAR to answer augmentation and rephrasing (ablation) | LLaMA-2-7B trained on GSM8K · MATH | accuracy: 4.4 -> 5.7 | Helped |
| Add rephrasing to answer augmentation (ablation) | LLaMA-2-7B trained on MATH · GSM8K | accuracy: 28.4 -> 29.1 | Helped |
| Add self-verification and FOBAR to answer augmentation and rephrasing (ablation) | LLaMA-2-7B trained on MATH · GSM8K | accuracy: 29.1 -> 34.6 | Helped |
| Add self-verification and FOBAR to answer augmentation and rephrasing (ablation) | LLaMA-2-7B trained on MATH · MATH | accuracy: 15.3 -> 17.7 | Helped |
| Add 20K answer-augmentation examples (ablation) | LLaMA-2-7B · GSM8K | accuracy: +0.1% | No clear effect |
| Add 20K rephrasing examples (ablation) | LLaMA-2-7B · GSM8K | accuracy: 0.4% | Helped |
| Add 20K FOBAR examples (ablation) | LLaMA-2-7B · GSM8K | accuracy: 2.3% | Helped |
| Add 20K self-verification examples (ablation) | LLaMA-2-7B · GSM8K | accuracy: 2.6% | Helped |
Surprising findings
- Training on reasoning paths with incorrect final answers still beats SFT on GSM8K (43.6 vs 41.6).
- Merging the external RFT dataset into MetaMathQA hurts performance.
Limitations
- Performance drops on longer GSM8K questions.
- MetaMath-70B trained with QLoRA rather than full finetuning due to compute limits.
- Relies on GPT-3.5-Turbo for generating rephrased questions and answers, so quality is bounded by the teacher.
Experimental setup
Datasets: GSM8K MATH
Benchmarks: GSM8K MATH DROP GSM8K-Backward Game of 24
Metrics: accuracy
Baselines: GPT-3.5-Turbo GPT-4 PaLM-2 Minerva WizardMath RFT SFT MAmmoTH Platypus Code-LLaMA
Builds on: Self-Verification FOBAR Rejection sampling fine-tuning (RFT) QLoRA Complexity-based CoT
Closest papers by meaning
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.91)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.90)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.90)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.90)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.90)
- Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.