Research map / Learning theory and algorithms / Prediction and decision losses
Scaling laws for learning with real and surrogate data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies weighted ERM that mixes a small set of real data with surrogate data, derives theory (Gaussian sequence, Sobolev regression, low-dim, high-dim ridge) and an empirical scaling law predicting test error as a function of the mixing weight alpha.
Problem
How to integrate scarce original (target-distribution) data with cheaper surrogate data (other sources or synthetic), how to weight them, and how many surrogate samples to add, with predictable test error.
Approach
Treat surrogate data as a regularizer in a weighted ERM with mixing weight alpha; optimally weighted ERM can beat training on original data even when surrogate distribution is unrelated (linked to Stein's paradox), and the test error follows a scaling law in n, m and alpha that can select alpha and the amount of surrogate data.
Key results
- Weighted ERM with optimal alpha beats training on original data alone even when the surrogate distribution is unrelated; in ridge regression with orthogonal parameters (gamma=pi/2) the test error still drops.
- A scaling law R(alpha) ≈ alpha^2 R_su(inf) + [alpha^2 (R_su(m)-R_su(inf))^{1/beta} + (1-alpha)^2 R_or(n)^{1/beta}]^beta predicts test error across alpha, n, m on Gaussian mixture, IMDB, CIFAR and TCGA experiments.
- Unweighted (naive) ERM's test error can degrade as surrogate data increases, while optimally weighted ERM is monotone decreasing in m and n.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add surrogate data using optimally tuned weighted ERM | Weighted ERM · Original-distribution test data | test error | Helped |
| Add surrogate data with optimally tuned weighting, even when its distribution is unrelated | Gaussian mean estimation · Original-distribution test data | test error | Helped |
| Use unweighted ERM instead of optimally weighted ERM | Training on original and surrogate data · Sentiment analysis | test error | Hurt |
| Increase surrogate data while using unweighted ERM | Training on original and surrogate data · Sentiment analysis | test error | Hurt |
| Add surrogate data with optimally tuned weighting | Ridge regression · High-dimensional simulated data, γ = π/2 | excess test risk | Helped |
| Use the proposed scaling law to predict mixture-training performance | Weighted ERM · Simulated and real-world datasets | test error prediction | Helped |
| Use validation data from the original distribution to select the surrogate weight | Weighted ERM · Original-distribution test data | test error | Helped |
Surprising findings
- Surrogate data can reduce test error even when unrelated to the original data (Stein's paradox effect).
- Naive unweighted pooling can make test error worse as the amount of surrogate data grows.
Limitations
- Scaling law assumes zero irreducible-gap beyond Bayes error and a fixed exponent beta; beta_or and beta_su can differ in practice.
- Theory is for simple models (Gaussian sequence, ridge, low-dimensional asymptotics) and does not directly cover classification loss.
- Alpha=1 showed discontinuity in the TCGA Cox experiment, requiring a proxy value of 0.95.
Experimental setup
Datasets: IMDB Rotten Tomatoes Goodreads book reviews CIFAR-10 CIFAR-100 TCGA PanCancer
Benchmarks: IMDB sentiment test split CIFAR-10 test set TCGA PFS survival
Metrics: test error excess risk 1-concordance score
Baselines: naive unweighted ERM training on original data only training on surrogate data only
Builds on: empirical risk minimization Stein's paradox data augmentation
Closest papers by meaning
- High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws (ICLR 2025 · similarity 0.88)
- Scaling Laws for the Value of Individual Data Points in Machine Learning (ICML 2024 · similarity 0.86)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.85)
- On the Asymptotic Distribution of the Minimum Empirical Risk (ICML 2024 · similarity 0.85)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.85)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.