Research map / Learning theory and algorithms / Prediction and decision losses
Distributionally Robust Data Valuation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes distributionally robust data valuation, using the marginal improvement of a chi-square-ball distributionally robust generalization error (DRGE), approximated by training-free NTK-based model deviation, to value data without a known validation distribution.
Problem
Data valuation typically relies on a known, fixed validation set/distribution, which is impractical in collaborative ML and data marketplaces where buyers' validation distributions differ and are unknown a priori.
Approach
Define data value via the marginal improvement of a worst-case (chi-square ball) generalization error, and use model deviation (RKHS norm difference of models with/without a point) as a proxy; for wide NNs, the NTK at initialization yields a training-free closed-form deviation, and low value is linked to low scarcity and dissimilarity (uniqueness).
Key results
- Deviation achieves the lowest DRGE for kernel regression on UBER, MNIST and CIFAR-10 when selecting 45% of data, and the lowest on all datasets except MNIST for NN.
- In data removal, the proposed method yields the fastest DRGE increase when removing high-value points first and the slowest when removing low-value points first in almost all scenarios.
- Theorem 3.5 bounds the NTK-at-initialization approximation error of model deviation, enabling a training-free valuation for NNs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Deviation instead of LOO for selecting data | Kernel regression · HOUSING | DRGE: 2.482(3.5e-02) -> 2.555(1.1e-01) | Hurt |
| Use Deviation instead of LOO for selecting data | Kernel regression · UBER | DRGE: 372.124(8.5e+00) -> 361.810(6.5e+00) | Helped |
| Use Deviation instead of LOO for selecting data | Kernel regression · DIABETES | DRGE: 0.908(5.7e-03) -> 1.057(4.3e-02) | Hurt |
| Use Deviation instead of TracIn for selecting data | Kernel regression · MNIST | DRGE: 1.974(3.5e-02) -> 1.893(6.5e-02) | Helped |
| Use Deviation instead of LOO for selecting data | Kernel regression · CIFAR-10 | DRGE: 2.479(1.7e-02) -> 2.470(2.3e-02) | Helped |
| Use Deviation instead of LOO for selecting data | Neural network · HOUSING | DRGE: 3.661(6.4e-02) -> 3.416(5.0e-02) | Helped |
| Use Deviation instead of LOO for selecting data | Neural network · UBER | DRGE: 269.047(7.3e+00) -> 265.654(9.3e+00) | Helped |
| Use Deviation instead of LOO for selecting data | Neural network · DIABETES | DRGE: 0.835(7.5e-03) -> 0.794(6.1e-03) | Helped |
| Use Deviation instead of CG for selecting data | Neural network · MNIST | DRGE: 0.200(1.5e-02) -> 0.224(1.4e-02) | Hurt |
| Use Deviation instead of Random for selecting data | Neural network · CIFAR-10 | DRGE: 0.122(3.3e-04) -> 0.122(6.4e-04) | No clear effect |
Surprising findings
- Several NN-based valuation methods (Influence, TracIn, CG, DAVINZ) and LAVA perform worse than Random on the kernel regression DRGE metric, since they optimize for performance on one specific validation distribution.
Limitations
- Theory requires wide NNs, small initial output, and large n for the bounds to be tight.
- Assumes buyers' validation distributions lie within a chi-square ball around the sampling distribution; DRGE is approximated using a very large sampling dataset.
- Experiments restricted to kernel regression and NN with LOO valuation; other learning algorithms not covered.
Experimental setup
Datasets: HOUSING UBER DIABETES MNIST CIFAR-10
Benchmarks: HOUSING UBER DIABETES MNIST CIFAR-10
Metrics: DRGE worst-case loss over 50 k-means validation clusters
Baselines: Random LOO Influence function TracIn LAVA CG DAVINZ
Builds on: Data Shapley leave-one-out valuation neural tangent kernel DAVINZ LAVA
Closest papers by meaning
- Data Distribution Valuation (NeurIPS 2024 · similarity 0.86)
- KAIROS: Scalable Model-Agnostic Data Valuation (NeurIPS 2025 · similarity 0.84)
- SAVA: Scalable Learning-Agnostic Data Valuation (ICLR 2025 · similarity 0.83)
- DeRDaVa: Deletion-Robust Data Valuation for Machine Learning (AAAI 2024 · similarity 0.83)
- GMValuator: Similarity-based Data Valuation for Generative Models (ICLR 2025 · similarity 0.83)
- DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.