Research map / Large language models / Instruction and fine-tuning
MAmmoTH2: Scaling Instructions from the Web
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mines ~10M instruction-response pairs from Common Crawl via fastText recall, LLM extraction, and LLM refinement (WebInstruct), then fine-tunes base LLMs into MAmmoTH2 models that gain strongly on held-out reasoning benchmarks.
Problem
Instruction tuning data for LLM reasoning is usually small and human-written or GPT-4 synthesized, which limits scale, diversity and quality; the paper asks whether high-quality instruction data already exists naturally in the web pre-training corpus.
Approach
A three-step pipeline (fastText-based recall of domain-relevant Common Crawl documents with GPT-3.5/4 domain filtering, LLM-based extraction of naturally occurring Q-A pairs, and LLM refinement that completes reasoning steps) harvests web instruction data without human annotation or GPT-4 distillation.
Key results
- MAmmoTH2-7B (Mistral base, WebInstruct only) raises MATH from 11.2% to 36.7% and GSM8K from 36.2% to 68.4% without in-domain training, averaging +14.0 points over Mistral-7B across seven benchmarks.
- MAmmoTH2-Plus-8B outperforms Llama-3-8B-Instruct by about 6 points on average on reasoning benchmarks (59.1 vs 53.4 AVG) with comparable general performance.
- MAmmoTH2-Plus-8x7B reaches an average of 62.9 on reasoning benchmarks, matching Qwen-1.5-110B (63.6) with 13B active parameters, and achieves AlpacaEval 2.0 win rate 33.8 and Arena Hard 32.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Mistral-7B on WebInstruct | Mistral-7B · Reasoning benchmarks (average) | accuracy: +14.0 AVG | Helped |
| Continue WebInstruct training with additional public instruction datasets (ablation) | Mistral-7B trained on WebInstruct · Reasoning benchmarks (average) | accuracy: 52.8 -> 58.0 | Helped |
| Merge data refined by Mixtral and Qwen (ablation) | Mistral-7B trained on refined data · GSM8K | accuracy: 62.9 / 65.4 -> 65.6 | Helped |
| Merge data refined by Mixtral and Qwen (ablation) | Mistral-7B trained on refined data · MATH | accuracy: 29.1 / 28.9 -> 31.0 | Helped |
| Use education-source data rather than forum-source data (ablation) | Mistral-7B · GSM8K | accuracy: 51.0 -> 58.0 | Helped |
| Use education-source data rather than forum-source data (ablation) | Mistral-7B · MATH | accuracy: 24.0 -> 24.8 | Helped |
| Train on Math-domain data (ablation) | Mistral-7B · MATH | accuracy: 15.7 -> 27.3 | Helped |
| Train on Math-domain data (ablation) | Mistral-7B · ARC-C | accuracy: 77.6 -> 74.1 | Hurt |
| Train on Science-domain data (ablation) | Mistral-7B · ARC-C | accuracy: 77.6 -> 83.6 | Helped |
| Train on diverse Other-domain data (ablation) | Mistral-7B · GSM8K | accuracy: 47.4 -> 59.4 | Helped |
| Continue WebInstruct training with additional public instruction datasets (ablation) | Mixtral-8x7B trained on WebInstruct · Reasoning benchmarks (average) | accuracy: 58.9 -> 62.9 | Helped |
Surprising findings
- Mistral-7B trained only on WebInstruct gains large held-out improvements on GSM8K (+32.2) despite no GSM-style in-domain training data.
- Training solely on the math domain did not improve science/STEM benchmarks, and the 'Other' domain gave the highest GSM8K score, suggesting diversity matters more than in-domain math.
Limitations
- Extracted data contains noise and inaccuracies; the refinement step can introduce hallucinations (about 10% in a 50-sample case study).
- Coverage is concentrated in math and science; humanities and daily chat topics are underrepresented.
- Quality depends on the performance of the open LLMs used for extraction and refinement, which may introduce biases.
Experimental setup
Datasets: WebInstruct OpenHermes 2.5 Code-Feedback Math-Plus (MetaMathQA, Orca-Math) Common Crawl GSM8K MATH TheoremQA BBH ARC-C GPQA MMLU-STEM
Benchmarks: TheoremQA MATH GSM8K GPQA MMLU-STEM BBH ARC-C HumanEval MBPP MMLU MMLU-Pro MT-Bench
Metrics: accuracy pass@1 win rate average score
Baselines: Mixtral-8x7B-Instruct Llama-3-8B-Instruct Qwen-1.5-110B Qwen-1.5-72B Deepseek-Math-Instruct WizardMath-7B-1.1 Mistral-7B-Instruct-v0.2 GPT-4-Turbo-0409
Builds on: fastText DeepSeekMath recall pipeline LLaMA-Factory DeepSpeed ZeRO MetaMathQA Orca-Math
Closest papers by meaning
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.92)
- Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.91)
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · similarity 0.91)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.91)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.91)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.