Research map / Large language models / Reasoning and chain of thought
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MAmmoTH, Llama-2 and Code Llama models instruction-tuned on MathInstruct (260K hybrid CoT/PoT math rationales) that generalize across in-domain and out-of-domain math benchmarks.
Problem
Open-source LLMs lag closed models on math reasoning, and dataset-specific fine-tuning (RFT, WizardMath) improves in-domain math but hurts out-of-domain generalization.
Approach
Combine broad-coverage math datasets with hybrid chain-of-thought and program-of-thought rationales (six newly GPT-4-curated) in one instruction-tuning mix, and decode PoT first with CoT fallback when the program is not executable.
Key results
- MAmmoTH-7B reaches 33.4% on MATH with Code Llama base (MAmmoTH-Coder), and 31.5% with Llama-2; the paper states it beats WizardMath-7B by 23 points on MATH.
- MAmmoTH-Coder-34B reaches 43.6% on MATH, which the paper says surpasses GPT-4 CoT (42.5%).
- Hybrid CoT+PoT training gives 47.9 average over nine datasets for Llama-2 7B, versus 32 (CoT-only) and 41 (PoT-only) subsets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hybrid CoT and PoT training instead of CoT-only training (ablation) | MAmmoTH-7B · All nine evaluation datasets | average accuracy: 32.0 -> 47.9 | Helped |
| Use hybrid CoT and PoT training instead of PoT-only training (ablation) | MAmmoTH-7B · All nine evaluation datasets | average accuracy: 41.0 -> 47.9 | Helped |
| Train on the CoT subset instead of WizardMath's dataset-specific data (ablation) | MAmmoTH-7B · All nine evaluation datasets | average accuracy: 27% to 32% | Helped |
| Train on the CoT subset instead of WizardMath's dataset-specific data (ablation) | MAmmoTH-7B · GSM8K and MATH | accuracy: 3% | Hurt |
| Train on the PoT subset instead of WizardMath's dataset-specific data (ablation) | MAmmoTH-7B · All nine evaluation datasets | overall accuracy: 27% to 41% | Helped |
| Use hybrid decoding rather than CoT decoding (ablation) | MAmmoTH-7B · All nine evaluation datasets | average accuracy: 33.0 -> 47.9 | Helped |
| Use hybrid decoding rather than PoT decoding (ablation) | MAmmoTH-7B · All nine evaluation datasets | average accuracy: 46.1 -> 47.9 | Helped |
| Add the six newly curated rationale datasets (ablation) | MAmmoTH-7B · All nine evaluation datasets | overall accuracy: 9% overall increase | Helped |
| Add diverse math datasets beyond GSM8K (ablation) | MAmmoTH-7B · All nine evaluation datasets | average accuracy: 22.7 -> 19.5 -> 19.2 -> -4.8 -> -3.4 -> 47.9 | Helped |
| Fine-tune Code-Llama rather than Llama-2, using MathInstruct | MAmmoTH-7B · All nine evaluation datasets | average accuracy: +4 | Helped |
Surprising findings
- PoT-only fine-tuning generally outperforms CoT-only fine-tuning, which the authors attribute to programs handling complex computation better.
- Dataset-specific tuning (WizardMath) improves in-domain GSM8K but underperforms on out-of-domain benchmarks such as AQuA and MMLU-Math.
Limitations
- Limited coverage of advanced fields such as mathematical analysis, complex analysis, graph theory, and numerical analysis.
- Not trained on proof-type problems, so theorem-proving ability is limited.
- Possible harmful or biased outputs; training data cleanliness not guaranteed.
Experimental setup
Datasets: MathInstruct GSM8K MATH AQuA-RAT TheoremQA Camel-Math MathQA NumGLUE
Benchmarks: GSM8K MATH AQuA-RAT NumGLUE SVAMP Mathematics SimulEq SAT-Math MMLU-Math
Metrics: accuracy
Baselines: WizardMath Platypus Code Llama (PoT) RFT GPT-4 PaLM-2 Claude-2 Galactica Vicuna-1.5 Tulu
Builds on: WizardMath Program-of-Thought PAL Self-Instruct Platypus
Closest papers by meaning
- OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset (NeurIPS 2024 · similarity 0.91)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.91)
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning (ICML 2024 · similarity 0.90)
- MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning (ICLR 2024 · similarity 0.90)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.90)
- OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.