Research map / Learning theory and algorithms / Prediction and decision losses
Better Training Data Attribution via Better Inverse Hessian-Vector Products
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASTRA, which uses the EKFAC GGN as a preconditioner for stochastic Neumann series iterations to compute more accurate iHVPs for gradient-based training data attribution (influence functions and SOURCE).
Problem
Gradient-based training data attribution (influence functions, unrolled differentiation) requires inverse Hessian-vector products, which are costly to approximate accurately; EKFAC is biased and cannot be improved with more compute, while stochastic Neumann series iterations (LiSSA) converge slowly
Approach
Use the EKFAC GGN eigendecomposition as a preconditioner for stochastic Neumann series iterations, so the iterative solver starts from the EKFAC solution and converges quickly to an accurate, consistent iHVP; the same approach is applied to SOURCE (ASTRA-SOURCE).
Key results
- ASTRA-IF reaches an LDS of around 0.5 across many settings, and ensembled ASTRA-IF about 0.6.
- On CIFAR-10 with ResNet-9, ensembled ASTRA-IF reaches about 0.6 versus 0.25 for EKFAC-IF.
- ASTRA usually converges in fewer than 200 iterations, while SNI makes slow progress and needs carefully tuned learning rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ensembled ASTRA-IF instead of EKFAC-IF | Influence functions on ResNet-9 · CIFAR-10 | LDS: 0.25 -> 0.6 | Helped |
| Replace EKFAC iHVP approximation with ASTRA | ASTRA-IF and ASTRA-SOURCE · UCI Concrete, Parkinsons, CIFAR-10, MNIST, FashionMNIST, FashionMNIST-N, and WikiText-2 | LDS | Helped |
| Ensemble predictions across model seeds | ASTRA TDA methods · Multiple experimental settings | LDS | Helped |
| Project influence estimates onto high-curvature eigenspaces, excluding lower-curvature directions (ablation) | Influence functions · MNIST and FashionMNIST | LDS | Hurt |
| Add low-curvature directions to influence estimates (ablation) | Influence functions · MNIST and FashionMNIST | LDS | Helped |
Surprising findings
- Low-curvature eigendirections are essential for influence function performance; projecting onto high-curvature subspaces degrades LDS substantially.
- Truncating Neumann series acts as an implicit damping that disproportionately affects low-curvature directions, so early stopping hurts influence quality.
Limitations
- ASTRA's gains depend on the damping hyperparameter; improvements over SNI are smaller at larger damping.
- LDS ground truth requires very expensive retraining (e.g., 1000 GPT-2 fine-tunes), and performance is only about 0.5 without ensembling.
Experimental setup
Datasets: UCI Concrete UCI Parkinsons CIFAR-10 MNIST FashionMNIST WikiText-2
Benchmarks: LDS on UCI Concrete LDS on UCI Parkinsons LDS on CIFAR-10 (ResNet-9) LDS on MNIST LDS on FashionMNIST LDS on FashionMNIST-N LDS on WikiText-2 (GPT-2)
Metrics: Linear Datamodeling Score (Spearman correlation) training loss vs iterations
Baselines: EKFAC-IF EKFAC-SOURCE SNI TracIn TRAK
Builds on: influence functions EKFAC LiSSA SOURCE TRAK
Closest papers by meaning
- Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods (NeurIPS 2025 · similarity 0.88)
- LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution (NeurIPS 2026 · similarity 0.86)
- Training Data Attribution via Approximate Unrolling (NeurIPS 2024 · similarity 0.86)
- Enhancing Training Data Attribution with Representational Optimization (NeurIPS 2025 · similarity 0.85)
- quanda: An Interpretability Toolkit for Training Data Attribution Evaluation (NeurIPS 2026 · similarity 0.84)
- Influence Functions for Scalable Data Attribution in Diffusion Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.