Research map / Learning theory and algorithms / Sample complexity
Data Distribution Valuation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an MMD-based method to value data distributions (not just datasets) under a Huber heterogeneity model, using the uniform vendor mixture as a reference when the ground truth is unknown.
Problem
How a data buyer should compare the value of heterogeneous vendor data distributions from small preview samples, without access to the ground-truth reference distribution.
Approach
Value a sampling distribution as the negated MMD to a reference distribution under a Huber heterogeneity model; replace the unknown reference with the uniform mixture of vendor distributions, justified by a game-theoretic worst-case argument, and derive finite-sample comparison margins.
Key results
- On CIFAR10 vs CIFAR100 with no validation set, Ours reaches Pearson 0.564 versus 0.481 for DAVINZ-style baselines and MMD^2 at 0.563 (rank correlation with test accuracy).
- On TON vs UGR16 with a validation set, Ours achieves Pearson 0.584, the highest among compared baselines.
- On CaliH vs KingH regression without a validation set, Ours achieves Pearson 0.767 versus 0.726 for MMD^2 and 0.048 for DAVINZ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use feature-label pairs and conditional distributions (ablation) | Ours on CIFAR10 vs. CIFAR100 · CIFAR10 vs. CIFAR100, D_val | Pearson correlation: 0.763(0.02) -> 0.989(0.01) | Helped |
| use MMD instead of MMD² | Ours on CIFAR10 vs. CIFAR100 · CIFAR10 vs. CIFAR100, D_U | Pearson correlation: 0.563(0.01) -> 0.564(0.02) | No clear effect |
| use MMD instead of MMD² | Ours on TON vs. UGR16 · TON vs. UGR16, D_U | Pearson correlation: 0.480(0.15) -> 0.461(0.14) | No clear effect |
| use MMD instead of MMD² | Ours on regression settings · CaliH vs. KingH and Census15 vs. Census17 | Pearson correlation: 0.872(0.03) -> 0.896(0.02); 0.889(0.05) -> 0.843(0.03) | Mixed |
| remove the explicit reference and use IG | Regression valuation · CaliH vs. KingH and Census15 vs. Census17 | Pearson correlation: -0.907(0.02); -0.932(0.02) | Hurt |
| remove the explicit reference and use VV | Regression valuation · CaliH vs. KingH and Census15 vs. Census17 | Pearson correlation: -0.603(0.01); -0.707(0.01) | Hurt |
| use MMD rather than DAVINZ | Ours on regression settings without D_val · CaliH vs. KingH and Census15 vs. Census17, D_U | Pearson correlation: 0.048(0.08) -> 0.767(0.04); 0.227(0.11) -> 0.769(0.08) | Helped |
Surprising findings
- Gradient-based valuation baselines LAVA and DAVINZ perform poorly under the Huber model, with LAVA's correlation flipping sign on CIFAR10 vs CIFAR100 once the validation set is replaced by the uniform mixture.
- Class-wise Shapley drops from 0.889 to -0.874 in correlation when the validation set is replaced by the uniform mixture.
Limitations
- Theoretical guarantees are limited to the Huber contamination model; non-Huber settings are only evaluated empirically.
- Heterogeneity model requires matched supports of P* and Q, restricting applicability to some datasets.
Experimental setup
Datasets: MNIST EMNIST FaMNIST CIFAR10 CIFAR100 Census15 Census17 CaliH KingH Credit7 Credit31 TON
Benchmarks: CIFAR10 vs CIFAR100 TON vs UGR16 CaliH vs KingH Census15 vs Census17 MNIST vs EMNIST
Metrics: Pearson correlation l2 error l_inf error pairwise inversions
Baselines: LAVA DAVINZ class-wise Shapley information-gain value volume value MMD^2
Builds on: Maximum mean discrepancy Huber contamination model Data Shapley LAVA DAVINZ
Closest papers by meaning
- Distributionally Robust Data Valuation (ICML 2024 · similarity 0.86)
- KAIROS: Scalable Model-Agnostic Data Valuation (NeurIPS 2025 · similarity 0.83)
- DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation (NeurIPS 2024 · similarity 0.83)
- An Instrumental Value for Data Production and its Application to Data Pricing (ICML 2025 · similarity 0.82)
- What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions (NeurIPS 2025 · similarity 0.82)
- Data Acquisition via Experimental Design for Data Markets (NeurIPS 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.