Research map / Large language models / Agents, code and math
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MathScale, which extracts topics and knowledge points from seed math questions, builds a concept graph, and random-walks it to prompt GPT-3.5 into generating 2M math QA pairs (MathScaleQA), used to instruction-tune LLaMA-2 and Mistral models evaluated on the new 10-dataset MwpBench.
Problem
Open-source LLMs lack mathematical reasoning ability, and existing math instruction data is small and augmentation-based methods produce examples too similar to seeds; evaluation is also inconsistent across math benchmarks.
Approach
Concept-graph-guided synthesis: extract topics and knowledge points from seed questions (concept compression), build a co-occurrence concept graph, sample concept compositions via random walks (connection forging), and prompt a frontier LLM to write new questions from them, enabling scaling to millions of examples.
Key results
- MathScale-7B reaches 35.0% micro and 37.5% macro average on MwpBench, surpassing best same-size peers by 42.9% and 43.7% relative.
- MathScale-Mistral reaches 38.7% micro and 40.8% macro average, on par with GPT-3.5-Turbo (39.8 / 41.5) on MwpBench.
- Performance on MwpBench grows roughly logarithmically with MathScaleQA size up to 2M examples for LLaMA-2 7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaMA-2 7B with MathScaleQA instead of MetaMath's training data | LLaMA-2 7B · MwpBench | micro average accuracy: 24.5 -> 35.0 | Helped |
| fine-tune LLaMA-2 7B with MathScaleQA instead of MetaMath's training data | LLaMA-2 7B · MwpBench | macro average accuracy: 26.1 -> 37.5 | Helped |
| remove 50% of the seed questions (ablation) | MathScale · MwpBench | macro average accuracy: 14.5 -> 14.0; -2.9% | Hurt |
| restrict seed questions to GSM8K and MATH training sets (ablation) | MathScale · MwpBench | macro average accuracy: 14.5 -> 13.9; -3.5% | Hurt |
| add a diverse set of seed questions (ablation) | MathScale · MwpBench | macro average accuracy: 13.9 -> 14.5; 14.0 -> 14.5 | Helped |
| remove 50% of the topics (ablation) | MathScale · MwpBench | macro average accuracy: 14.5 -> 14.1; -2.3% | Hurt |
| add the full set of topics rather than removing 50% (ablation) | MathScale · MwpBench | macro average accuracy: 14.1 -> 14.5; -2.3% | Helped |
| remove 50% of the knowledge points (ablation) | MathScale · MwpBench | macro average accuracy: 14.5 -> 13.2; -8.6% | Hurt |
| retain the full set of knowledge points rather than removing 50% (ablation) | MathScale · MwpBench | macro average accuracy: 13.2 -> 14.5; -8.6% | Helped |
| replace 26% of GPT-3.5 solutions with GPT-4-corrected solutions (ablation) | MathScale data generation · MwpBench | micro average accuracy: 10.6 -> 10.2 | Hurt |
| replace 26% of GPT-3.5 solutions with GPT-4-corrected solutions (ablation) | MathScale data generation · MwpBench | macro average accuracy: 11.5 -> 11.1 | Hurt |
| replace all GPT-3.5 solutions with GPT-4 solutions (ablation) | MathScale data generation · MwpBench | micro average accuracy: 10.6 -> 9.8 | Hurt |
Surprising findings
- Validating and correcting GPT-3.5 solutions with GPT-4 did not improve results, suggesting distillation benefits persist despite some incorrect solutions.
- Removing half the knowledge points hurt more (-8.6% macro) than removing half the topics (-2.3%).
Limitations
- Data generated by GPT-3.5 may contain incorrect solutions and possible test-set leakage into GPT-3.5 or LLaMA-2 training data.
- Only natural-language reasoning; tool integration left for future work.
- Did not scale to LLaMA-2 70B or training sets beyond 2M examples.
Experimental setup
Datasets: GSM8K MATH TAL-SCQ Math23k Ape210k GaokaoBench-Math AGIEval CollegeMath
Benchmarks: MwpBench GSM8K MATH CollegeMath TAL-SCQ Math23k Ape210k GaokaoBench-Math AGIEval-Gaokao-Math AGIEval-SAT-Math AGIEval-Math Fresh-GaokaoMath-2023
Metrics: accuracy micro average accuracy macro average accuracy
Baselines: WizardMath MetaMath MAmmoTH GAIR-Abel GPT-3.5-Turbo GPT-4 LLaMA-2
Builds on: Alpaca open-instruct WizardMath (evol-instruct) MetaMath
Closest papers by meaning
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.91)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.91)
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (ICLR 2024 · similarity 0.90)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.90)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.