Research map / Learning theory and algorithms / Prediction and decision losses
Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice
Jiachen T. Wang, Tong Wu, Kaifeng Lyu, James Zou, Dawn Song, Ruoxi Jia, Prateek Mittal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train proxy model with tiny learning rate (below 1e-4) instead of standard learning rate | proxy models GPT2-Small, Pythia-70M, OPT-125M · Pile validation loss ranking of 23 recipes | Spearman rank correlation: < 0.75 -> > 0.92 | Improved |
| train proxy with tiny learning rate | proxy models · downstream benchmarks and Pile domains | rank correlation | Improved |
| train proxy with tiny learning rate | proxy models · top-k decision regret on Pile val loss | validation loss gap: > 0.25 -> near zero | Improved |
| increase learning rate slightly from lower to higher value | GPT2-Small on DCLM vs DCLM-dedup-GS · Pile validation loss and 5 downstream tasks | dataset ranking | Hurt |
| tune proxy learning rate within near-optimal range instead of a fixed one | proxy dataset comparison · DCLM vs stricter dedup variant | ranking stability | Hurt |
Closest papers by meaning
- Small-to-Large Generalization: Training Data Influences Models Consistently Across Scale (ICLR 2025 · similarity 0.87)
- Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · similarity 0.87)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.87)
- A Bitter Lesson for Data Filtering (NeurIPS 2026 · similarity 0.87)
- DataDecide: How to Predict Best Pretraining Data with Small Experiments (ICML 2025 · similarity 0.87)
- DataRater: Meta-Learned Dataset Curation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
