Research map / Learning theory and algorithms / Prediction and decision losses
Distributional Training Data Attribution: What do Influence Functions Sample?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces distributional training data attribution (d-TDA), predicting how the distribution of trained-model outputs changes under dataset edits, and shows influence functions arise as limits of unrolled differentiation and optimal transport maps between Boltzmann distributions.
Problem
Classical training data attribution assumes deterministic training, ignoring stochasticity from initialisation and batching, so it cannot separate effects of dataset changes from sampling randomness.
Approach
Frame TDA as predicting changes in the distribution over trained models (distributional influence), and show that influence functions are asymptotic samples of unrolled differentiation under SGD with local-minimum convergence and a KL-optimal transport map for low-temperature Boltzmann distributions, without convexity assumptions.
Key results
- Unrolled differentiation response correlates about 90% with influence-function predictions at late training times on UCI Concrete regression.
- On CIFAR-10 with a Swin Transformer, distributional-TDA data pruning of 5000 points yields greater average test accuracy gains than fixed-seed IF pruning.
- Theorem shows influence functions minimise asymptotic KL divergence among local transport maps for low-temperature Boltzmann distributions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use distributional influence for data pruning instead of fixed-seed influence | SWIN Vision Transformer · CIFAR-10 | test accuracy | Helped |
| Train longer | Unrolled differentiation response compared with influence functions · UCI Concrete | correlation: 90% | Helped |
| Use a lower step size | Unrolled differentiation compared with influence functions · UCI Concrete | correlation | Helped |
| Evaluate attribution distributionally rather than with fixed-seed TDA | Influence functions · CIFAR-10 | test accuracy | Helped |
| Use Wasserstein influence instead of mean influence to identify influential examples | Distributional influence · Synthetic 1D regression | variance after removing an example | Helped |
| Use unrolled differentiation to predict leave-one-out retraining distributions | Unrolled differentiation · Leave-one-out retraining | correlation of empirical means | Helped |
Surprising findings
- Leave-one-out correlation of TDA methods is near zero per model, but the means of predicted and true measurement distributions correlate strongly, suggesting low signal-to-noise rather than method failure.
- Influence functions remain valid for non-convex losses with degenerate local minima under the stated assumptions.
Limitations
- Convergence guarantees require training to reach local minima with fast gradient-flow convergence (assumption A6) and long training times.
- Hessian nullspace component of the unrolled response is not captured by influence functions, though empirically small for tested tasks.
- Distributional LDS benchmark is only preliminary.
Experimental setup
Datasets: CIFAR-10 UCI Concrete ArtBench-10 MNIST
Benchmarks: CIFAR-10 ArtBench-10 UCI Concrete
Metrics: test accuracy validation loss Wasserstein distance Spearman correlation (distributional LDS) Pearson correlation of mean changes
Baselines: influence functions (fixed-seed TDA) mean influence approximate unrolled differentiation
Builds on: influence functions unrolled differentiation linear datamodelling score
Closest papers by meaning
- LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution (NeurIPS 2026 · similarity 0.87)
- Influence Functions for Scalable Data Attribution in Diffusion Models (ICLR 2025 · similarity 0.87)
- Training Data Attribution via Approximate Unrolling (NeurIPS 2024 · similarity 0.86)
- Rescaled Influence Functions: Accurate Data Attribution in High Dimension (NeurIPS 2025 · similarity 0.86)
- Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods (NeurIPS 2025 · similarity 0.85)
- Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.