Research map / Learning theory and algorithms / Prediction and decision losses
KAIROS: Scalable Model-Agnostic Data Valuation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KAIROS, a model-agnostic data valuation method that scores each training example by its closed-form MMD-based distributional influence (plus a conditional MCMD term for labels), enabling fast online updates.
Problem
Data valuation that is both model-agnostic and scalable: model-based methods depend on one fitted model, Data Shapley requires many retrainings, and Wasserstein-based approaches (e.g., LAVA) produce non-unique or approximate influence rankings and are costly online.
Approach
Define each training point's value as its directional-derivative influence on MMD between the training distribution and a clean validation distribution; MMD yields a closed-form influence (kernel mean differences) requiring no retraining, extended to labels with a conditional MMD term computed from a classifier's predicted probabilities, and updated incrementally for streaming batches.
Key results
- KAIROS ranks top-100 points with 99% overlap to exact LOO ranking, versus 60% for LAVA (per Figure 1 claim).
- In online setting on CIFAR-10 (100 batches of 100), KAIROS reaches up to 50x speedup over LAVA and 28x over KNNShapley.
- KAIROS attains top-1 detection accuracy in 6 of 8 noise-detection scenarios and top-2 in the remaining 2; LAVA needs about 9K validation samples to reach 0.77 detection accuracy on IMDB where KAIROS needs 30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KAIROS MMD influence instead of Lava Wasserstein influence | Data valuation · LOO ranking, top 100 points | overlap with true LOO set: 60% -> 99% | Helped |
| Use KAIROS instead of Lava | IMDB feature-noise detection · IMDB feature-noise detection | detection accuracy: 9K validation samples -> 30 validation samples to reach 0.77 | Helped |
| Use KAIROS online updates instead of recalculating values with Lava | CIFAR-10 online data valuation · CIFAR-10 online data valuation, 100 batches | speedup: 50x | Helped |
| Use KAIROS online updates instead of recalculating values with KNNShapley | CIFAR-10 online data valuation · CIFAR-10 online data valuation, 100 batches | speedup: 28x | Helped |
| Use KAIROS instead of baseline methods for feature-noise ranking | Noise detection · Feature-noise detection across four datasets | corrupted-sample detection performance | Helped |
| Use KAIROS instead of baseline methods for label-noise ranking | Noise detection · Label-noise detection, especially AG News and STL-10 | corrupted-sample detection performance | Helped |
| Use KAIROS instead of other methods for malicious-data ranking | Poisoning detection · Badnet, poison frog, style, and Lisa attacks | poison detection performance | Mixed |
| Remove the least-valuable points ranked by KAIROS | Test accuracy after data pruning · STL-10 with 20% mislabels | test accuracy | Helped |
| Remove the most-valuable points ranked by KAIROS | Test accuracy after data pruning · STL-10 and IMDB with 20% mislabels | test accuracy drop | Helped |
| Use KAIROS instead of Lava for offline valuation | CIFAR-10 data valuation · CIFAR-10 offline runtime | runtime | No clear effect |
Surprising findings
- Directly valuing on the joint (X,Y) product kernel can reduce sensitivity to label noise in high dimensions, motivating a separate conditional term.
- Authors report LAVA's Sinkhorn approximation rankings deviate from even the true Sinkhorn LOO rankings.
Limitations
- Uses fixed kernels and a fixed balancing coefficient (0.03) across all tasks
- Finite-sample separation may fail for small bandwidths due to high variance
- Requires a classifier trained on clean validation data for the conditional term
Experimental setup
Datasets: CIFAR-10 STL-10 IMDB AG News
Benchmarks: CIFAR-10 STL-10 IMDB AG News
Metrics: cumulative fraction of corrupted data recovered detection accuracy test accuracy after point removal runtime
Baselines: LAVA DataOOB DVRL KNNShapley
Builds on: LAVA Data Shapley OpenDataVal MCMD
Closest papers by meaning
- SAVA: Scalable Learning-Agnostic Data Valuation (ICLR 2025 · similarity 0.87)
- One Sample Fits All: Approximating All Probabilistic Values Simultaneously and Efficiently (NeurIPS 2024 · similarity 0.86)
- GMValuator: Similarity-based Data Valuation for Generative Models (ICLR 2025 · similarity 0.85)
- Convex Dataset Valuation for Post-Training (ICML 2026 · similarity 0.85)
- What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions (NeurIPS 2025 · similarity 0.85)
- Distributionally Robust Data Valuation (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.