Research map / Large language models / Language, safety and interpretability

Improving Pretraining Data Using Perplexity Correlations

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Selects pretraining data by estimating per-domain correlations between web-corpus bits-per-byte losses of ~90 public LLMs and downstream benchmark performance, then filtering with a fastText classifier, without training any LLMs.

Problem

Data selection for LLM pretraining is expensive because it normally requires many proxy model training runs; the paper asks whether existing public models can be used observationally to select high-quality pretraining data.

Approach

Replace unobserved data-mixture sampling probabilities with observed per-domain log-likelihoods of public LLMs, estimate a rank-based correlation between each domain's loss and benchmark error, and project these correlations onto a valid, repetition-limited sampling distribution (plus a fastText classifier) to choose pretraining pages.

Key results

• At 160M scale over 8 evals, perplexity correlations achieve average rank 1.750, versus 1.375 for handcrafted fastText plus manual language filter and 4.500 for DSIR.
• Perplexity correlations outperform DSIR on every benchmark and match the DCLM fastText filter (with English filter) at 160M scale.
• In preregistered experiments up to 1.4B parameters, benefits increase with scale on the raw DCLM pool, but there is little benefit on the already heavily filtered pool.

Measured effects reported in the paper

Surprising findings

• On the extensively pre-filtered DCLM pool, perplexity correlations gave no clear benefit, since correlation coefficients there were nearly uniformly positive.
• Ranking of models by predicted performance works well for unusual architectures such as Mamba, but poorly for models trained on unusual data such as Phi.

Limitations

• Predictions are weaker for LLMs trained on atypical data (e.g., Phi), suggesting the model pool or data pool must be more diverse.
• Theory relies on Gaussian single-index model assumptions that deviate from real LLM loss data.
• Gains vanish on already extensively filtered pretraining pools.

Experimental setup

Datasets: RedPajama V2 sample RedPajama V2 sample-100B DCLM data pools LAMBADA ARC Easy PIQA SciQ Open LLM Leaderboard

Benchmarks: LAMBADA ARC Easy PIQA SciQ LAMBADA IT LAMBADA FR LAMBADA DE LAMBADA ES DCLM Core (22 benchmarks) Non-EN LAMBADA

Metrics: accuracy log perplexity average rank R^2 (5-fold leave-out)

Baselines: DSIR DCLM fastText classifier English language filter uniform sampling language filtering

Builds on: DSIR DataComp-LM datamodeling fastText

Closest papers by meaning

• Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.91)
• An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.91)
• Fine-tuning can Help Detect Pretraining Data from Large Language Models (ICLR 2025 · similarity 0.91)
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.89)
• Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models (ICLR 2025 · similarity 0.89)
• Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.89)