Research map / Large language models / Instruction and fine-tuning
MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MASS, which builds a math skill graph from NuminaMath using LLM-extracted skills and uses it to score and select high-quality subsets of math web and synthetic corpora for continual pretraining of TinyLlama-1.1B and Mistral-7B.
Problem
Existing data selection methods for LLM pretraining focus on general domains and overlook mathematical skill structure, so selecting math pretraining data that is efficient and effective remains under-addressed.
Approach
Extract mathematical skills from a high-quality reference set with an LLM, build a skill co-occurrence graph, score target documents by semantic similarity to skills aggregated over the graph (diagonal importance plus compositional edges), and keep the top-ranked fraction.
Key results
- On Mistral-7B with OpenWebMath-pro, MASS reaches 62.0 average vs 56.1 for the unselected data at the same trained tokens (+5.9%).
- On TinyLlama-1.1B, MASS selected subsets match original-data performance with 43%-71% fewer trained tokens and outperform by 3.2%-5.9% at equal tokens.
- On TinyLlama-1.1B with Jiuzhang3.0, MASS averages 40.8 vs 36.4 for the original data and 38.3 for AutoDS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MASS-selected data instead of original data | TinyLlama-1.1B · OpenWebMath | Avg.: +3.6% | Helped |
| Use MASS-selected data instead of original data | TinyLlama-1.1B · OpenWebMath-pro | Avg.: +3.3% | Helped |
| Use MASS-selected data instead of original data | TinyLlama-1.1B · Jiuzhang3.0 | Avg.: +4.2% | Helped |
| Use MASS-selected data instead of original data | Mistral-7B · OpenWebMath | Avg.: +4.8% | Helped |
| Use MASS-selected data instead of original data | Mistral-7B · OpenWebMath-pro | Avg.: +5.9% | Helped |
| Use MASS-selected data instead of original data | Mistral-7B · Jiuzhang3.0 | Avg.: +4.1% | Helped |
| Add diagonal adjacency-matrix weights for individual-skill importance (ablation) | MASS · OpenWebMath-pro | Avg.: 26.5 -> 26.9; 0.4%↓ | Helped |
| Add diagonal adjacency-matrix weights for individual-skill importance (ablation) | MASS · Jiuzhang3.0 | Avg.: 39.6 -> 40.8; 1.2%↓ | Helped |
| Add non-diagonal adjacency-matrix weights for skill composition (ablation) | MASS · OpenWebMath-pro | Avg.: 24.1 -> 26.9; 2.8%↓ | Helped |
| Add non-diagonal adjacency-matrix weights for skill composition (ablation) | MASS · Jiuzhang3.0 | Avg.: 37.2 -> 40.8; 2.1%↓ | Helped |
| Use maximum cosine similarity rather than mean embeddings (ablation) | MASS · OpenWebMath-pro | Avg.: 26.4 -> 26.9; 0.5%↓ | Helped |
| Use maximum cosine similarity rather than mean embeddings (ablation) | MASS · Jiuzhang3.0 | Avg.: 37.4 -> 40.8; 3.4%↓ | Helped |
Surprising findings
- Jiuzhang3.0 synthetic data, trained on only 6.8B tokens, outperforms web datasets trained on about 15B tokens, which the authors attribute to synthetic data quality.
- Removing the diagonal (skill importance) terms of the adjacency matrix hurts less than removing the non-diagonal (compositional) terms, though both matter.
Limitations
- Skill extraction quality is not filtered against a taxonomy, so extracted skills may be noisy.
- Relies on skill frequency as importance and ignores skill difficulty and diversity.
- Baselines were mainly implemented only for TinyLlama-1.1B due to compute constraints.
Experimental setup
Datasets: NuminaMath OpenWebMath OpenWebMath-pro Jiuzhang3.0
Benchmarks: GSM8K MATH SVAMP ASDiv MAWPS TAB MQA MMLU STEM SAT MATH
Metrics: accuracy average accuracy
Baselines: AutoDS DSIR RHO-1 ProX BM25 FineWeb heuristic rules RULE
Builds on: AutoDS DSIR RHO-1 ProX Skill-it
Closest papers by meaning
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.90)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.89)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.89)
- MathPile: A Billion-Token-Scale Pretraining Corpus for Math (NeurIPS 2024 · similarity 0.89)
- Neuro-Symbolic Data Generation for Math Reasoning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.