Research map / Learning theory and algorithms / Prediction and decision losses
A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Versatile Influence Function (VIF), a finite-difference approximation of the statistical influence function that extends IF-based data attribution to non-decomposable losses (Cox partial likelihood, DeepWalk contrastive loss, ListMLE) using auto-differentiation.
Problem
Influence-function data attribution has been limited to losses that decompose into a sum of per-data-point losses (M-estimators), excluding contrastive, ranking, and survival losses where a loss term depends on multiple data points.
Approach
Define a finite-difference influence by perturbing the empirical distribution toward the leave-one-out distribution, which is equivalent to evaluating the loss with object-presence vector b; the resulting VIF is the inverse Hessian times the gradient of the difference L(θ,1) − L(θ,1₋ᵢ), computed with auto-differentiation and requiring no retraining.
Key results
- On Cox regression, VIF reaches Pearson correlation 0.997 (METABRIC) and 0.943 (SUPPORT) against brute-force LOO (0.997 and 0.955 for retraining seeds).
- VIF is up to 1097x faster than brute-force LOO retraining (SUPPORT: 225 min vs 12.3 sec).
- On listwise learning-to-rank, VIF correlates 0.823 (Mediamill) and 0.906 (Delicious) with brute-force LOO, whose reference correlation is 0.999.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIF instead of brute-force LOO retraining | Cox regression · METABRIC | runtime: 593$\times$ | Helped |
| Use VIF instead of brute-force LOO retraining | Cox regression · SUPPORT | runtime: 1097$\times$ | Helped |
| Use VIF instead of brute-force LOO retraining | DeepWalk node embedding · Karate | runtime: 1.87$\times$ | Helped |
| Use VIF instead of brute-force LOO retraining | Linear listwise ranker · Mediamill | runtime: 20$\times$ | Helped |
| Use VIF instead of brute-force LOO retraining | Linear listwise ranker · Delicious | runtime: 236$\times$ | Helped |
| Use VIF to estimate influence rather than brute-force LOO retraining | Cox regression · METABRIC | Pearson correlation with brute-force LOO: 0.997 vs 0.997 | No clear effect |
| Use VIF to estimate influence rather than brute-force LOO retraining | Cox regression · SUPPORT | Pearson correlation with brute-force LOO: 0.943 vs 0.955 | No clear effect |
| Use VIF to estimate influence rather than brute-force LOO retraining | DeepWalk node embedding · Karate | Pearson correlation with brute-force LOO: 0.407 vs 0.419 | No clear effect |
| Use VIF to estimate influence rather than brute-force LOO retraining | Linear listwise ranker · Mediamill | Pearson correlation with brute-force LOO: 0.823 vs 0.999 | Hurt |
| Use VIF to estimate influence rather than brute-force LOO retraining | Linear listwise ranker · Delicious | Pearson correlation with brute-force LOO: 0.906 vs 0.999 | Hurt |
Surprising findings
- On Mediamill, VIF's correlation with brute-force LOO (0.823) is notably lower than the brute-force reference (0.999), suggesting weaker approximation for ranking losses.
- On the Karate node-embedding task, VIF speed-up is only 1.87x because random-walk overhead dominates.
Limitations
- Formal derivation assumes a strictly convex loss; applying VIF to neural networks requires heuristic tricks such as CG or LiSSA.
- Approximation quality is not guaranteed for general non-decomposable losses; approximation error is only bounded theoretically for Cox regression.
- Node-embedding and randomized losses (negative sampling, random walks) reduce correlation with LOO retraining.
Experimental setup
Datasets: METABRIC SUPPORT Zachary's Karate network Delicious Mediamill
Benchmarks: METABRIC SUPPORT Zachary's Karate network Delicious Mediamill
Metrics: Pearson correlation with brute-force LOO retraining runtime speed-up ratio
Baselines: brute-force leave-one-out retraining
Builds on: influence functions (Koh & Liang) robust statistics influence function DeepWalk ListMLE
Closest papers by meaning
- Rescaled Influence Functions: Accurate Data Attribution in High Dimension (NeurIPS 2025 · similarity 0.91)
- Influence Functions for Scalable Data Attribution in Diffusion Models (ICLR 2025 · similarity 0.89)
- LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution (NeurIPS 2026 · similarity 0.88)
- Towards Robust Influence Functions with Flat Validation Minima (ICML 2025 · similarity 0.87)
- Capturing the Temporal Dependence of Training Data Influence (ICLR 2025 · similarity 0.87)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.