Research map / Large language models / Language, safety and interpretability
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that perplexity from a 125M reference model can prune pretraining data (The Pile, Dolma) to improve downstream performance and training efficiency of 1B and 3B MPT models, including over-trained and data-constrained regimes.
Problem
Whether small language models can select high-quality subsets of large pretraining datasets to improve much larger LLMs, and how this depends on dataset domain composition, training regime, and evaluation metric.
Approach
Train a 125M-parameter reference model on a split of the data, score every sample by its perplexity, keep samples in a chosen perplexity range (low, medium, or high), and train larger MPT models on the pruned data; evaluate on downstream tasks rather than test-set perplexity.
Key results
- Pruning The Pile with high-perplexity selection at 50% raised 3B model average normalized accuracy by 2.04 (18.63 to 20.67).
- Perplexity-pruned 1B and 3B Pile models reach baseline performance in 1.31x and 1.45x fewer steps.
- Pruned Dolma data lowered test-set perplexity (13.53 to 14.33) while improving downstream average (13.84 to 15.35), showing test perplexity can be misleading.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select high-perplexity samples at 50% rate instead of using all data | MPT-1B trained on Pile · Pile | average normalized accuracy: 13.73 -> 15.62; 1.89 | Helped |
| Select high-perplexity samples at 50% rate instead of using all data | MPT-3B trained on Pile · Pile | average normalized accuracy: 18.63 -> 20.67; 2.04 | Helped |
| Select medium-perplexity samples at 50% rate instead of using all data | MPT-1B trained on Dolma · Dolma | average normalized accuracy: 13.84 -> 15.35; 1.51 | Helped |
| Select medium-perplexity samples at 50% rate instead of using all data | MPT-3B trained on Dolma · Dolma | average normalized accuracy: 19.2 -> 19.79; 0.59 | Helped |
| Select high-perplexity data when training 1B models for 5× the Chinchilla budget | MPT-1B trained on Pile · Pile, 5× Over-Trained | average normalized accuracy over baseline: 1.74 | Helped |
| Select medium-perplexity data when training 1B models for 5× the Chinchilla budget | MPT-1B trained on Dolma · Dolma, 5× Over-Trained | average normalized accuracy over baseline: 0.84 | Helped |
| Use perplexity-pruned data rather than unpruned data | MPT-1B trained on Pile · Pile, pretraining test split | test set perplexity: 7.83 -> 8.51 | Hurt |
| Use perplexity-pruned data rather than unpruned data | MPT-1B trained on Dolma · Dolma, pretraining test split | test set perplexity: 13.53 -> 14.33 | Hurt |
| Select medium-perplexity rather than unpruned data (ablation) | MPT-1B trained on Pile · Pile | average normalized accuracy: 13.73 -> 13.5; 0.23 | Hurt |
| Select low-perplexity rather than unpruned data (ablation) | MPT-1B trained on Pile · Pile | average normalized accuracy: 13.73 -> 9.56 | Hurt |
| Select low-perplexity rather than unpruned data (ablation) | MPT-1B trained on Dolma · Dolma | average normalized accuracy: 13.84 -> 12.96 | Hurt |
| Select high-perplexity rather than unpruned data (ablation) | MPT-1B trained on Dolma · Dolma | average normalized accuracy: 13.84 -> 14.52 | Helped |
Surprising findings
- The best selection criteria on one dataset (medium perplexity on Dolma) degrades performance when applied to the Pile (-0.23 vs baseline).
- Pruned data gains shrink for Dolma in the 5x over-trained regime (gain 0.84 vs 1.51 at Chinchilla optimal).
Limitations
- Only one reference model size (125M) and two datasets were studied.
- Pruning gains shrink or are absent in the over-trained regime for Dolma.
- Optimal selection criteria did not transfer across dataset compositions, so pruning settings must be tuned per dataset.
Experimental setup
Datasets: The Pile Dolma
Benchmarks: MMLU HellaSwag ARC easy ARC challenge LAMBADA PIQA SQuAD BoolQ MosaicML evaluation gauntlet (33 tasks)
Metrics: average normalized accuracy (random-guess normalized) test set perplexity training steps to baseline performance
Baselines: No pruning baseline Low perplexity selection Medium perplexity selection High perplexity selection
Builds on: MPT Chinchilla scaling laws decoupled Lion llm-foundry MosaicML evaluation gauntlet
Closest papers by meaning
- BaWA: Automatic Optimizing Pruning Metric for Large Language Models with Balanced Weight and Activation (ICML 2025 · similarity 0.89)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.89)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.89)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.89)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.89)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.