Research map / Learning theory and algorithms / Prediction and decision losses

Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining

NeurIPS 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Active learning and subset selection · 0 citations (OpenAlex)

Read the paper Check your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Large-scale study of hyperparameter sensitivity of data attribution methods (IF variants, TracIn, TRAK, LoGra), plus a theory-driven retraining-free surrogate indicator for selecting the influence-function regularization λ.

Problem

Hyperparameters of data attribution methods are under-explored, and standard evaluation metrics (LOO, LDS) require many subset retrainings, making hyperparameter tuning prohibitively expensive.

Approach

Derives a population Pearson LDS proxy and a sufficient condition for its monotonicity in the regularization λ, yielding a surrogate indicator computed from the full-data model alone; choose λ where the indicator's validation average crosses 0.5.

Key results

Increasing regularization from 1e-4 to 1e-2 for IF (CG) yields about a 3x increase in LDS.
The proposed surrogate-indicator selection yields LDS significantly better than lambda=0 and is often close to optimal across six IFFIM/TRAK settings.
No single fixed spectrum quantile for the baseline heuristic consistently performs well.

Measured effects reported in the paper

What was changed	Where	Measured	Effect
Increase regularization from 1e-4 to 1e-2	IF (CG) · MNIST+MLP	LDS: 3× increase	Helped
Set training epoch to 5	TRAK · MNIST+MLP and CIFAR-2+ResNet-9	LDS	Mixed
Increase projection dimension while tuning regularization	TRAK · Multiple experimental settings	LDS	Helped
Use high projection dimension with low regularization	TRAK · Multiple experimental settings	LDS	Hurt
Select regularization with the surrogate-indicator procedure instead of setting λ=0	IFFIM · Multiple experimental settings	LDS	Helped
Select regularization with the surrogate-indicator procedure instead of a fixed spectrum quantile	IFFIM and TRAK · Multiple experimental settings	LDS	Helped
Use the surrogate-indicator procedure to select regularization	TRAK · Various experimental settings with different projection dimensions	LDS	Helped
Use a larger training epoch for attribution scoring	Multiple data attribution methods · Multiple experimental settings	LDS	Hurt

Surprising findings

Larger training epoch of the checkpoint used for attribution can lead to worse LDS, counter-intuitively.
The apparent decline of TRAK LDS with higher projection dimension is due to insufficient regularization tuning rather than an intrinsic effect.

Limitations

Theory and selection focus only on the regularization term of influence-function-style methods.
Evaluation relies primarily on LDS; other metrics not examined.
Some theoretical assumptions do not always strictly hold, per the authors.

Experimental setup

Datasets: MNIST CIFAR-2 WikiText2 MAESTRO

Benchmarks: LDS on MNIST+MLP LDS on CIFAR-2+ResNet-9 LDS on WikiText2+GPT2 LDS on MAESTRO+MusicTransformer data selection downstream model performance drop

Metrics: Linear Datamodeling Score performance drop after removing top influential data

Baselines: spectrum-based quantile heuristic for lambda lambda=0

Builds on: influence functions TRAK LoGra TracIn linear datamodeling score

Closest papers by meaning

$\texttt{dattri}$: A Library for Efficient Data Attribution (NeurIPS 2024 · similarity 0.88)
Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods (NeurIPS 2025 · similarity 0.88)
quanda: An Interpretability Toolkit for Training Data Attribution Evaluation (NeurIPS 2026 · similarity 0.87)
Enhancing Training Data Attribution with Representational Optimization (NeurIPS 2025 · similarity 0.86)
Capturing the Temporal Dependence of Training Data Influence (ICLR 2025 · similarity 0.86)
DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models (NeurIPS 2025 · similarity 0.85)

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft