Research map / Large language models / Language, safety and interpretability
Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PreSelect, a lightweight document-level data selection method that scores pretraining documents by how well their normalized losses across six Llama models rank-predict downstream performance, then trains a fastText scorer to select data.
Problem
Pretraining data selection typically relies on human heuristics or expensive influence/training-based methods; the paper seeks an efficient, principled way to estimate each document's contribution to learning downstream abilities.
Approach
Define a document's "predictive strength" as how well the ranking of models' normalized losses on that document matches their downstream benchmark ranking, use high-scoring vs. low-scoring documents as positives/negatives to train a fastText scorer, and select pretraining data with it; uses document-level granularity and a small set of Llama-1/2 models instead of many diverse models.
Key results
- At 1B scale on RefinedWeb, PreSelect trained on 30B tokens reaches 33.4 average on selected tasks versus 31.3 for random trained on 300B tokens, indicating roughly 10x compute reduction.
- Outperforms DCLM by about 2 points on average at 1B/30B (33.4 vs 31.2) and at 3B/100B (39.5 vs 37.8).
- On C4 with Pythia architecture at 1B/25B tokens, PreSelect averages 46.8 versus 44.7 for random and 45.8 for MATES.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select documents using PreSelect instead of random sampling | 400M NEO pretraining on RefinedWeb · RefinedWeb, 8B tokens | Avg. accuracy: 24.2 -> 27.0 | Helped |
| Select documents using PreSelect instead of random sampling | 1B NEO pretraining on RefinedWeb · RefinedWeb, 30B tokens | Avg. accuracy: 28.1 -> 33.4 | Helped |
| Select documents using PreSelect instead of DCLM | 1B NEO pretraining on RefinedWeb · RefinedWeb, 30B tokens | Avg. accuracy: 31.2 -> 33.4 | Helped |
| Select documents using PreSelect instead of DCLM | 3B NEO pretraining on RefinedWeb · RefinedWeb, 100B tokens | Avg. accuracy: 37.8 -> 39.5 | Helped |
| Select documents using PreSelect instead of random sampling | 410M Pythia pretraining on C4 · C4, 25B tokens | Average accuracy: 42.7 -> 44.9 | Helped |
| Select documents using PreSelect instead of random sampling | 1B Pythia pretraining on C4 · C4, 25B tokens | Average accuracy: 44.7 -> 46.8 | Helped |
| Replace PreSelect's multi-model predictive-strength data scoring with ScalingFilter using only Llama-65B and Llama-7B (ablation) | 1B NEO pretraining on RefinedWeb · RefinedWeb, 30B tokens | Average benchmark performance: 40.3 -> 37.6 | Hurt |
| Use HellaSwag rather than average benchmark performance to rank models and select data (ablation) | PreSelect at 400M · Knowledge-intensive tasks; wiki-related raw text | Loss: 5% lower losses on wiki-related raw texts; 8% on math; 16% on code | Mixed |
Surprising findings
- Perplexity-correlation-based domain selection (DD) gave only marginal gains over random and dropped performance on some tasks such as HellaSwag, attributed to lost diversity.
- A ScalingFilter-style perplexity-difference metric selected short, easy data and gave only 37.6 vs 37.2 for random, with near-zero rank correlation to PreSelect's predictive strength.
Experimental setup
Datasets: RefinedWeb C4
Benchmarks: MMLU ARC-Easy ARC-Challenge HellaSwag PIQA SIQA SciQ RTE BBH LAMBADA OpenBookQA RACE
Metrics: accuracy exact-match bits per character
Baselines: Random PPL Filtering FineWeb-Edu PPL Correlation (DD) PPL Correlation (DP) DCLM ScalingFilter DSIR DsDm QuRating MATES
Builds on: Perplexity Correlation DCLM FineWeb-Edu ScalingFilter MATES
Closest papers by meaning
- MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models (NeurIPS 2024 · similarity 0.92)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.91)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.91)
- Data Selection via Optimal Control for Language Models (ICLR 2025 · similarity 0.90)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.90)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.