Research map / Learning theory and algorithms / Prediction and decision losses
Scaling Laws for the Value of Individual Data Points in Machine Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes individualized data scaling laws showing each training point's marginal contribution decays log-linearly with dataset size with point-specific exponents, plus likelihood-based and amortized estimators, applied to data valuation and point addition.
Problem
Aggregate scaling laws describe error vs. dataset size but ignore how the value of individual data points changes with dataset size; fitting per-point scaling behavior is costly.
Approach
Model each data point's expected marginal contribution as c(z)/k^alpha(z), with point-specific exponent and coefficient; fit it efficiently via a heteroscedastic Gaussian likelihood over few noisy samples, or amortize across points with a neural network predicting the parameters from the input.
Key results
- Log-linear scaling law fits marginal contributions with overall R^2 of 0.987 for logistic regression on IMDB, and 0.929-0.963 across SVM/MLP/logistic regression settings.
- Amortized estimator achieves Pearson correlation 0.994 on MiniBooNE for data valuation with 10 samples per point, versus 0.967 for Monte Carlo.
- Points selected by scaling laws at k=100 improve test accuracy by 3.28% on CIFAR-10 at k=100 but perform poorly at k=1000, showing the most valuable points depend on dataset size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use amortized estimator instead of per-point likelihood estimator (10 samples per point) (ablation) | data valuation scores · MiniBooNE, CIFAR-10, IMDB | Pearson correlation with ground truth: MiniBooNE 0.858→0.994; CIFAR-10 0.077→0.893; IMDB 0.342→0.930 | Helped |
| Parametric likelihood estimator with only 10 samples per data point (ablation) | data valuation scores · CIFAR-10, IMDB, MiniBooNE | Pearson correlation with ground truth: 0.077 (CIFAR-10) and 0.342 (IMDB) vs Monte Carlo 0.857 and 0.916 | Hurt |
| Scaling-law-based valuation vs Monte Carlo at 10 samples per point | data valuation scores (amortized estimator) · MiniBooNE, CIFAR-10, IMDB | Pearson correlation with ground truth: Amortized better on MiniBooNE (0.994 vs 0.967), IMDB (0.930 vs 0.916), and CIFAR-10 (0.893 vs 0.857) | Mixed |
| Scaling-law estimators with 50 samples per point vs Monte Carlo | data valuation scores · CIFAR-10 | Pearson correlation with ground truth: Likelihood 0.936 and Amortized 0.931 vs Monte Carlo 0.965 | Hurt |
| Select points to add using predicted contribution at the current dataset size (Scaling 100 vs Scaling 1000) for k=100 (ablation) | point addition to dataset of size 100 · MiniBooNE, CIFAR-10, IMDB | Accuracy improvement (%): Scaling 100 beats Random and Shapley on all three (e.g., CIFAR-10 3.28 vs 2.01 random); Scaling 1000 is 1.99 on CIFAR-10 | Helped |
| Select points using scaling predictions for the wrong dataset size (Scaling 100 at k=1000, Scaling 1000 at k=100) (ablation) | point addition · preceding dataset sizes 100 and 1000 | Accuracy improvement (%): Scaling 100 at k=1000 gives 0.06 on IMDB vs Scaling 1000 0.13; Scaling 1000 at k=100 gives 0.88 on IMDB vs Scaling 100 2 | Hurt |
| Select points by Distributional Shapley vs Scaling 1000 at k=1000 | point addition to dataset of size 1000 · MiniBooNE, CIFAR-10, IMDB | Accuracy improvement (%): Shapley and Scaling 1000 similar (e.g., MiniBooNE 0.22 vs 0.20; CIFAR-10 0.18 vs 0.17) | No clear effect |
Surprising findings
- Most valuable data points change with dataset size: high-contribution points at small k are often different from those at large k, so ranking depends on k.
- Pearson correlation of estimated alpha(z) across model types is weak for MiniBooNE (near zero for LogReg/MLP) though positive for CIFAR-10 and IMDB.
Limitations
- Fitting per-point scaling requires separate marginal contribution samples for each point, which remains expensive.
- Scaling-based selection ignores interactions with the contents of the current dataset, so selecting many top points may be suboptimal.
- Experiments are limited to simple models and small-scale datasets; theory relies on strong regularity assumptions and does not give precise predictions.
Experimental setup
Datasets: MiniBooNE CIFAR-10 IMDB
Benchmarks: MiniBooNE CIFAR-10 IMDB
Metrics: R^2 Pearson correlation accuracy improvement cross-entropy loss
Baselines: Monte Carlo Distributional Shapley Random selection
Builds on: Distributional Data Shapley Chinchilla scaling laws influence functions
Closest papers by meaning
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.87)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.87)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.87)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.86)
- Scaling laws for learning with real and surrogate data (NeurIPS 2024 · similarity 0.86)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.