Research map / Optimization / Optimizers
Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes training data attribution in the final-model-only setting as measuring model sensitivity via averaged further training, and unifies gradient-based TDA methods (Grad-Dot/Cos, influence functions, TRAK, EK-FAC, DataInf) as approximations to it.
Problem
Training data attribution when only the final trained model is available (no training algorithm or intermediate checkpoints), where there is no clear gold standard for evaluating attribution methods.
Approach
Define a further-training gold standard (leave-one-out further training with mean adjustment and averaging over training randomness) for sensitivity, and show via first- and second-order Taylor expansions that first-order (Grad-Dot/Cos) and influence-function methods (CG, LiSSA, EK-FAC, TRAK, DataInf) approximate it, deriving generalized influence functions for non-convex, non-stationary settings.
Key results
- First-order methods (Grad-Dot) reach among the highest initial cosine similarity to further training, but this decays with the amount of further training.
- Influence function methods (CG, LiSSA) are more stable over further training but never reach the peak cosine similarity of first-order methods.
- Averaging over more random seeds (up to 100) consistently increases cosine similarity to the gold scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Grad-Dot instead of influence-function methods | FiMODA attribution on the evaluated datasets · Tabular, CIFAR-10, and SST-2 | cosine similarity to further-training attribution | Mixed |
| Increase the amount of further training | Grad-Dot and Grad-Cos · Evaluated datasets | cosine similarity to further-training attribution | Hurt |
| Use DataInf instead of Grad-Dot | FiMODA attribution · Evaluated datasets | cosine similarity to further-training attribution | Helped |
| Use LiSSA-H instead of CG or LiSSA | Influence-function attribution · Evaluated datasets | cosine similarity to further-training attribution | Hurt |
| Use EK-FAC instead of LiSSA | Influence-function attribution · Evaluated datasets | cosine similarity to further-training attribution | Mixed |
| Average further-training runs over more random seeds | Further-training gold attribution · Evaluated datasets | maximum cosine similarity to gradient-based attribution | Helped |
| Subtract the mean effects across leave-one-out datasets rather than the full-dataset further-training effect | Further-training attribution adjustment · Some evaluated datasets | noise in cosine similarity | Helped |
| Set CG and LiSSA damping to 0.01 instead of 0.001 (ablation) | CG and LiSSA · Evaluated datasets | cosine similarity to further-training attribution | Helped |
Surprising findings
- Influence function methods, usually considered stronger in TAA/CPA settings, underperform simple first-order gradient methods in the FiMO setting at low further-training amounts.
- TRAK with a single checkpoint (M=1) gave poor approximations to further training.
Limitations
- The further training gold standard is computationally expensive, limiting scale of models and number of left-out instances.
- Experiments are limited to 50-100 training instances and test points, and gradient methods underperform on CIFAR-10 and SST-2.
Experimental setup
Datasets: Concrete Strength Energy Efficiency FICO Challenge Folktables CIFAR-10 SST-2
Benchmarks: Concrete Strength Energy Efficiency FICO Challenge Folktables CIFAR-10 SST-2
Metrics: cosine similarity to further-training attribution scores Spearman correlation (appendix)
Baselines: Grad-Dot Grad-Cos CG influence LiSSA LiSSA-H TRAK (M=1) EK-FAC DataInf
Builds on: influence functions TracIn TRAK DataInf EK-FAC
Closest papers by meaning
- quanda: An Interpretability Toolkit for Training Data Attribution Evaluation (NeurIPS 2026 · similarity 0.90)
- LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution (NeurIPS 2026 · similarity 0.89)
- Training Data Attribution via Approximate Unrolling (NeurIPS 2024 · similarity 0.89)
- Better Training Data Attribution via Better Inverse Hessian-Vector Products (NeurIPS 2025 · similarity 0.88)
- Training Data Attribution in Diffusion Models via Mirrored Unlearning and Noise-Consistent Skew (NeurIPS 2026 · similarity 0.88)
- Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.