Research map / Learning theory and algorithms / Prediction and decision losses
Rescaled Influence Functions: Accurate Data Attribution in High Dimension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes rescaled influence functions (RIF), a drop-in, additive data-attribution estimator that rescales per-sample influence by leave-one-out leverage-like factors to approximate single Newton steps, improving accuracy over influence functions in high-dimensional logistic regression.
Problem
Influence functions underestimate leave-T-out effects and are imprecise when the number of parameters is comparable to or exceeds the number of samples, even for logistic regression.
Approach
Define each sample's attribution as the single Newton step for leaving that sample out, using the leave-one-out Hessian; for GLMs and ReLU nets this equals the IF multiplied by a leverage-based rescale factor (1-h_i)^-1, keeping additivity and near-IF cost.
Key results
- RIF predictions closely track ground-truth retraining effects across all datasets, metrics and removal fractions (0.1% to 5%), while IF typically underestimates the effect.
- On IMDB with varying n/d, IF degrades as n approaches d while RIF stays accurate.
- RIF adds negligible overhead: rescaling takes 0.1-0.7% of total runtime across datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace influence functions with rescaled influence functions | Logistic regression with frozen-embedding features · DogFish, CatDog, AutoTruck, Enron, IMDB, and ESC-50; leave-T-out prediction, test loss, and self-loss | leave-T-out effect prediction accuracy | Helped |
| Replace influence functions with rescaled influence functions | Logistic regression with frozen-embedding features · Truck vs Automobile poisoned-sample removal | prediction effect estimation accuracy | Helped |
| Increase the sample-to-dimension ratio | Influence functions versus rescaled influence functions on IMDB · IMDB, self-loss leave-T-out estimates | prediction accuracy | Mixed |
| Increase L2 regularization | Influence functions versus rescaled influence functions on DogFish · DogFish, self-loss leave-T-out estimates | prediction accuracy | Mixed |
| Use RIF instead of IF for leave-T-out estimation | Logistic regression with frozen-embedding features · CatDog, leave-T-out effect across subset-selection strategies | effect ordering accuracy | Helped |
Surprising findings
- Influence-function underestimation and the correlation between IF and ground truth can be explained by per-sample rescale factors that are roughly constant (greater than 1), so IF is a linear distortion of RIF.
- IF can fail even to order subsets correctly on the Cat vs Dog dataset, whereas RIF remains accurate.
Limitations
- RIF is still imperfect on some settings such as ESC-50 and rightmost IMDB variants.
- Evaluated mainly on high-dimensional logistic regression; the authors expect it to be a model organism for broader ML.
- RIF still requires Hessian inversion, which may be infeasible at large scale; RIF is not a secure general defense against poisoning.
Experimental setup
Datasets: ESC-50 CIFAR-10 DogFish Enron spam IMDB
Benchmarks: ESC-50 CatDog AutoTruck DogFish Enron IMDB
Metrics: predicted vs actual leave-T-out effect on test predictions, accuracy of predicted effect of poisoned sample removal
Baselines: influence functions Newton step Additive-One-Exact data model retraining ground truth
Builds on: influence functions Newton step leave-one-out approximation TRAK infinitesimal jackknife
Closest papers by meaning
- A Versatile Influence Function for Data Attribution with Non-Decomposable Loss (ICML 2025 · similarity 0.91)
- Influence Functions for Scalable Data Attribution in Diffusion Models (ICLR 2025 · similarity 0.88)
- Towards Robust Influence Functions with Flat Validation Minima (ICML 2025 · similarity 0.86)
- LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution (NeurIPS 2026 · similarity 0.86)
- Distributional Training Data Attribution: What do Influence Functions Sample? (NeurIPS 2025 · similarity 0.86)
- Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.