Research map / Label-efficient and robust learning / Recognition and long-tail
Small-to-Large Generalization: Training Data Influences Models Consistently Across Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that training-data effects on language model losses are highly correlated across scale (proxy models down to ~370x less compute than 760M reference), and characterizes how proxy scale affects data attribution (TRAK/LDS on ImageNet and CIFAR-10) and DsDm dataset selection.
Problem
Proxy (small-scale) models are standard for estimating how training data changes large-model behavior, but model behavior changes across scale, so it is unclear when proxies are effective.
Approach
Empirically measures correlation between small proxy and 760M reference model losses across many training data distributions, then tests how proxy scale affects downstream data attribution (datamodels via TRAK) and dataset selection (DsDm).
Key results
- 57M proxy model losses correlate highly with 760M reference model losses across 10 training distributions while using ~175x less compute.
- In ImageNet data attribution, LDS drops by at most ~10% (0.21 to 0.19) even for proxies 1,000x smaller than the reference model.
- DsDm data selection using proxies down to 4x less compute than the 760M reference still substantially improves over random selection on SQuAD and LAMBADA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a 57M proxy LM instead of the 760M reference LM | GPT-2-style language models trained on varied data distributions · SQuAD, HellaSwag, LAMBADA, TriviaQA, C4, and the Pile | correlation of proxy and reference losses across training distributions | Helped |
| Reduce proxy model scale toward 40M parameters | 760M reference LM · SQuAD and TriviaQA | correlation of proxy and reference losses across training distributions | Hurt |
| Use a 57M proxy LM | 760M reference LM · HellaSwag and COPA | correlation with reference-model losses | Helped |
| Use smaller proxy ResNets for TRAK data attribution | Largest ResNet · ImageNet-1k | LDS: 0.21 to 0.19 | Hurt |
| Use proxy models more than 1,000× smaller than the reference model | Largest ResNet · CIFAR-10 | LDS | Hurt |
| Select pretraining data with DsDm attribution scores instead of choosing a random subset | 760M reference LM · SQuAD and LAMBADA | accuracy | Helped |
| Reduce proxy training scale to 4× less compute than the reference model | DsDm dataset selection for a 760M reference LM · SQuAD and LAMBADA | accuracy | No clear effect |
| Drop the Q matrix term from the TRAK estimator (ablation) | TRAK data attribution · Vision and language-model attribution settings | LDS | Helped |
Surprising findings
- Proxy models can correlate highly with the reference model on HellaSwag and COPA even when their accuracy is at or below random guessing.
- Only very small proxies (e.g., 40M, ~370x less compute) predict large-model behavior poorly, notably on SQuAD and TriviaQA.
Limitations
- LDS values are small in absolute terms (peak ~0.21 on ImageNet), so exact prediction is not achieved.
- Reference model limited to 760M parameters due to academic compute budget.
- Results depend on specific datamodel estimator (TRAK) and may change with better estimators.
Experimental setup
Datasets: MPT C4 The Pile SQuAD LAMBADA HellaSwag TriviaQA COPA ImageNet-1k CIFAR-10 CIFAR-100 Wikipedia
Benchmarks: C4 The Pile SQuAD LAMBADA HellaSwag TriviaQA COPA ImageNet-1k CIFAR-10 CIFAR-100
Metrics: loss correlation (R^2 / Spearman) Linear Datamodeling Score (LDS) downstream accuracy
Baselines: DsDm with 760M reference model random subset selection
Builds on: datamodels TRAK DsDm Chinchilla scaling laws
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.89)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.88)
- DataDecide: How to Predict Best Pretraining Data with Small Experiments (ICML 2025 · similarity 0.88)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.88)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.