Research map / Large language models / Language, safety and interpretability
MathPile: A Billion-Token-Scale Pretraining Corpus for Math
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MathPile, a ~9.5B-token math-centric pretraining corpus built from textbooks, arXiv, Wikipedia, ProofWiki, StackExchange and Common Crawl with math-specific cleaning, deduplication and benchmark contamination removal, and shows continual pretraining of Mistral-7B gives gains on math benchmarks.
Problem
Open-source, high-quality, large-scale pretraining corpora for mathematical reasoning are scarce, since prominent math corpora (Minerva, MathMix) are not released and existing open ones are narrow or contaminated.
Approach
Assembles a diverse math-centric corpus from multiple curated sources with source-specific preprocessing, customized per-source language-ID thresholds, math-specific heuristic filters, MinHash deduplication, and early benchmark contamination detection, then validates it via continual pretraining of Mistral-7B under a "less is more" quality principle.
Key results
- Continual pretraining of Mistral-7B on the Textbooks subset (0.56B tokens) raises MATH from 10.08 to 12.10 and SAT-MATH from 47.27 to 56.36.
- Training on the full mix of Textbooks, Wikipedia, StackExchange and CC (4B tokens) lifts GSM8K from 47.38 to 49.88 and MATH from 10.08 to 11.70.
- Training on AMPS and DM-Mathematics subsets degrades performance, with DM-Mathematics scoring 0 on all benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continual-pre-train on MathPile Textbooks | Mistral-7B-v0.1 · SAT-MATH | accuracy: 47.27 -> 56.36 | Helped |
| Continual-pre-train on MathPile Textbooks | Mistral-7B-v0.1 · MATH | accuracy: 10.08 -> 12.10 | Helped |
| Continual-pre-train on MathPile Wikipedia | Mistral-7B-v0.1 · GSM8K | accuracy: 47.38 -> 49.96 | Helped |
| Continual-pre-train on MathPile Wikipedia | Mistral-7B-v0.1 · AQuA | accuracy: 27.95 -> 35.43 | Helped |
| Continual-pre-train on MathPile StackExchange | Mistral-7B-v0.1 · GSM8K | accuracy: 47.38 -> 43.06 | Hurt |
| Continual-pre-train on MathPile arXiv | Mistral-7B-v0.1 · MATH | accuracy: 10.08 -> 7.50 | Hurt |
| Continual-pre-train on MathPile arXiv | Mistral-7B-v0.1 · GSM8K | accuracy: 47.38 -> 47.91 | Helped |
| Continual-pre-train on MathPile Textbooks, Wikipedia, StackExchange, and Common Crawl | Mistral-7B-v0.1 · MATH | accuracy: 10.08 -> 11.70 | Helped |
| Continual-pre-train on synthetic AMPS data | Mistral-7B-v0.1 · GSM8K | accuracy: 47.38 -> 0.08 | Hurt |
| Continual-pre-train on synthetic DM-Mathematics data | Mistral-7B-v0.1 · GSM8K | accuracy: 47.38 -> 0.00 | Hurt |
| Continual-pre-train on sampled OpenWebMath | Mistral-7B-v0.1 · GSM8K | accuracy: 47.38 -> 43.21 | Hurt |
| Skip global data processing and use sampled raw Wikipedia (ablation) | Mistral-7B-v0.1 · GSM8K | accuracy: 47.38 -> 41.92 | Hurt |
Surprising findings
- Pretraining on synthetic AMPS and DM-Mathematics datasets degraded performance, with DM-Mathematics scoring 0 on all benchmarks.
- Pretraining on arXiv subset lowered MATH, SAT-MATH and MathQA scores despite its size.
Limitations
- Many cleaning and filtering decisions were made empirically and may not be optimal.
- Data scale is insufficient for extra-large models; Common Crawl subset could be expanded.
- Corpus is focused primarily on English.
Experimental setup
Datasets: MathPile OpenWebMath AMPS DM-Mathematics SlimPajama GSM8K MATH MMLU-STEM AGIEval-SAT-MATH MathQA AQuA
Benchmarks: GSM8K MATH AGIEval-SAT-MATH MMLU-Math MathQA AQuA
Metrics: accuracy
Baselines: OpenWebMath AMPS DM-Mathematics Mistral-7B-v0.1
Builds on: RedPajama SlimPajama OpenWebMath NaturalProofs
Closest papers by meaning
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.89)
- OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (ICLR 2024 · similarity 0.89)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.89)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.88)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.88)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.