Research map / Label-efficient and robust learning / Noisy labels
Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that training amortized models on cheap unbiased noisy labels ("stochastic amortization") accurately approximates Shapley, Banzhaf, LIME, Data Shapley and distributional data valuation, often with an order-of-magnitude speedup.
Problem
Per-example computation of explainable ML quantities (feature attribution, data valuation) is intractable for large datasets, and standard amortization requires expensive exact labels that are often infeasible to generate.
Approach
Train amortized networks with noisy labels from unbiased Monte Carlo or sampling estimators instead of exact labels; theory shows bias causes learning the wrong function while variance only slows optimization, and unbiased noise leaves the regression objective equivalent up to a constant.
Key results
- For Distributional Data Shapley on CIFAR-10 with 5 Monte Carlo samples per point, amortization reaches 0.81 correlation with ground truth versus 0.58 for Monte Carlo estimates.
- For Shapley feature attribution on ImageNette, amortized error is similar to running KernelSHAP for 10-40K samples although labels use an order of magnitude fewer samples.
- For tabular Data Shapley, amortized predictions achieve correlation above 0.9 with ground truth using only 50 Monte Carlo samples per data point on MiniBooNE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Amortize noisy Data Shapley estimates instead of using the estimates directly | Data valuation on MiniBooNE · MiniBooNE | Pearson correlation with ground truth: correlation >0.9 when using only 50 samples | Helped |
| Amortize distributional Data Shapley estimates instead of using Monte Carlo estimates directly | Distributional data valuation · CIFAR-10 | Pearson correlation with ground truth: 0.58 -> 0.81 | Helped |
| Use more training data for amortization | Data valuation · Adult census and MiniBooNE | Pearson correlation with ground truth: 250 to 10K data points | Helped |
| Use an unbiased estimator for Shapley value attribution labels | Stochastic amortization · ImageNette | Attribution estimation error: 10-40K samples | Helped |
| Use SGD-Shapley labels, which have non-negligible bias, instead of unbiased Shapley estimators (ablation) | Stochastic amortization · ImageNette | Attribution estimation error | Hurt |
| Amortize KernelSHAP estimates rather than compute attributions per example | Shapley value feature attribution · ImageNette | Attribution estimation error: datasets ranging from 250-10K data points | Helped |
| Amortize Banzhaf value estimates | Stochastic amortization · Feature attribution | Attribution estimation error | Helped |
| Amortize LIME estimates | Stochastic amortization · Feature attribution | Attribution estimation error | Helped |
| Use more training examples for distributional valuation amortization | Distributional data valuation · CIFAR-10 | Pearson correlation with ground truth: 5K data points | Helped |
| Use amortized distributional valuation scores to identify mislabeled examples instead of Monte Carlo estimates | Distributional data valuation · CIFAR-10 | Mislabeled-example identification reliability | Helped |
Surprising findings
- Biased labels (SGD-Shapley) lead to poor amortization, while variance in unbiased labels mainly slows optimization rather than changing the learned function.
Limitations
- Very high label noise can prevent effective optimization.
- Hard to know a priori how much compute label generation needs to reach a target error.
- Amortization is less effective for Banzhaf values and LIME due to inconsistent attribution scales across inputs.
Experimental setup
Datasets: ImageNette adult census MiniBooNE CIFAR-10
Benchmarks: ImageNette feature attribution adult census data valuation MiniBooNE data valuation CIFAR-10 distributional data valuation
Metrics: squared error Pearson correlation Spearman correlation estimation error vs FLOPs
Baselines: KernelSHAP permutation sampling SGD-Shapley FastSHAP TMC estimator Monte Carlo estimates
Builds on: KernelSHAP FastSHAP TMC-Shapley Datamodels
Closest papers by meaning
- Selective Explanations (NeurIPS 2024 · similarity 0.86)
- Fast Data Attribution for Text-to-Image Models (NeurIPS 2025 · similarity 0.84)
- Amortized Linear-time Exact Shapley Value for Product-Kernel Methods (NeurIPS 2026 · similarity 0.84)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.84)
- Diffusion Models for Attribution (AAAI 2025 · similarity 0.83)
- Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled Yet Hard-to-Learn Samples in Noisy Data (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.