Research map / Large language models / Language, safety and interpretability
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenWebMath, a 14.7B-token dataset of English mathematical web pages extracted from Common Crawl with LaTeX-preserving extraction, filtering, and deduplication, and shows 1.4B models trained on it outperform models trained on far more general tokens on math perplexity and accuracy.
Problem
Public web datasets strip or mangle mathematical notation (LaTeX), so the benefits of large-scale math web pretraining seen in Minerva are unavailable to the open research community.
Approach
A custom pipeline that recovers LaTeX from MathJax delimiters, alttext, images and MathML in Common Crawl HTML, combined with a fastText-based MathScore classifier trained on LaTeX-stripped text, language-ID and perplexity filters, and SimHash deduplication to produce an open math web corpus.
Key results
- Models trained on 14.7B OpenWebMath tokens reach GSM8k perplexity 1.9075 versus 1.9430 for Pythia 1.4B trained on 300B Pile tokens, and a 50/50 ProofPile+OpenWebMath mixture reaches 1.8968.
- On MATH Algebra-Easy, the OpenWebMath-trained 1.4B model achieves 5.62% accuracy versus 3.93% for Pythia 1.4B (300B tokens).
- On LILA multiarith, the OpenWebMath model reaches 16.67% versus 9.77% for The Pile trained on the same 14.7B tokens, though Pythia 300B reaches 21.80%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on OpenWebMath instead of The Pile (300B tokens) | 1.4B language model · GSM8k | perplexity: 1.9430 -> 1.9075 | Helped |
| Train on a 50/50 ProofPile and OpenWebMath mixture instead of OpenWebMath alone | 1.4B language model · GSM8k | perplexity: 1.9075 -> 1.8968 | Helped |
| Train on OpenWebMath instead of The Pile (300B tokens) | 1.4B language model · MATH | perplexity: 1.7117 -> 1.6285; 1.7560 -> 1.6503; 1.6358 -> 1.5949; 1.6359 -> 1.6002; 1.7460 -> 1.6894; 1.5191 -> 1.4542; 1.7252 -> 1. | Helped |
| Train on a 50/50 ProofPile and OpenWebMath mixture instead of OpenWebMath alone | 1.4B language model · MATH | perplexity: 1.6285 -> 1.6055; 1.6503 -> 1.6190; 1.5949 -> 1.5301; 1.6002 -> 1.5719; 1.6894 -> 1.6607; 1.4542 -> 1.4119; 1.5748 -> 1. | Helped |
| Train on OpenWebMath instead of The Pile (300B tokens) | 1.4B language model · MATH Algebra-Easy | accuracy: 3.93% -> 5.62% | Helped |
| Train on a 50/50 ProofPile and OpenWebMath mixture instead of OpenWebMath alone | 1.4B language model · MATH Algebra-Easy maj@16 | accuracy: 9.55% -> 10.11% | Helped |
| Train on OpenWebMath instead of The Pile (300B tokens) | 1.4B language model · LILA multiarith | accuracy: 21.80% -> 16.67% | Hurt |
| Train on a 50/50 ProofPile and OpenWebMath mixture instead of OpenWebMath alone | 1.4B language model · LILA multiarith | accuracy: 16.67% -> 13.22% | Hurt |
Surprising findings
- On LILA multiarith, Pythia 1.4B trained on 300B Pile tokens (21.80%) outperforms the OpenWebMath-trained model (16.67%), so the 20x-token advantage does not hold uniformly across all benchmarks.
- Common Crawl HTML lacks JavaScript-rendered MathJax, so the authors found very few of the standard LaTeX script tags reported by prior work and had to detect dollar-sign delimiters themselves.
Limitations
- Pipeline was run only once, so most design choices lack empirical ablation.
- English-only dataset, limiting multilingual use.
- Prefilter may be too aggressive, and only text of math is kept without associated figures.
Experimental setup
Datasets: Common Crawl The Pile ProofPile MATH GSM8k LILA multiarith
Benchmarks: GSM8k perplexity MATH perplexity MATH Algebra-Easy LILA multiarith
Metrics: perplexity accuracy maj@16 accuracy
Baselines: The Pile ProofPile Pythia 1.4B
Builds on: Minerva The Pile ProofPile Resiliparse Pythia
Closest papers by meaning
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.90)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.89)
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (ICLR 2024 · similarity 0.89)
- MathPile: A Billion-Token-Scale Pretraining Corpus for Math (NeurIPS 2024 · similarity 0.89)
- Llemma: An Open Language Model for Mathematics (ICLR 2024 · similarity 0.89)
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.