Research map / Learning theory and algorithms / Prediction and decision losses
Towards Robust Influence Functions with Flat Validation Minima
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a flat-validation-minima influence function (FVM) that computes influence on a SAM-tuned validation model with a second-order loss-change term, improving mislabeled-sample detection and relabeling over standard IF methods.
Problem
Influence functions give unreliable sample-influence estimates in deep networks, especially on noisy training data, because loss-change estimation fails when the validation risk is sharp.
Approach
Theoretically bounds influence-sign estimation error by validation risk and its sharpness, then computes influence at a flat validation minimum (via SAM-style optimization) using a second-order loss-change term and a diagonal Fisher preconditioner for the inverse Hessian.
Key results
- On CIFAR-10N Worst, FVM reaches 96.46 ROC AUC and 94.97 AP for mislabel detection, versus 65.75 ROC AUC for LiSSA.
- On CIFAR-100N relabeling, FVM achieves 70.61% top-1 accuracy versus 22.41% for GEX (best baseline).
- On CIFAR-10N Worst relabeling, FVM reaches 86.09% vs 80.35% for GEX, and FVM gains about 12 points over VM on CIFAR-10N Aggre relabeling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FVM instead of VM | Influence estimation · CIFAR-10N Worst mislabeled detection | ROC AUC: 95.88$\pm$0.13 -> 96.46$\pm$0.08 | Helped |
| use FVM instead of VM | Influence estimation · CIFAR-10N Worst mislabeled detection | AP: 94.27$\pm$0.21 -> 94.97$\pm$0.12 | Helped |
| use FVM instead of VM | Influence estimation · CIFAR-10N Aggre relabeling | Top-1 relabeling accuracy: 94.17$\pm$0.02 -> 94.63$\pm$0.04 | Helped |
| use FVM instead of VM | Influence estimation · CIFAR-100N Noisy relabeling | Top-1 relabeling accuracy: 58.13$\pm$0.15 -> 70.61$\pm$0.24 | Helped |
| use FVM instead of VM | Influence estimation · Sentence Transformations | ROC AUC: 99.97$\pm$0.16 -> 99.92$\pm$0.18 | Hurt |
| use FVM instead of VM | Influence estimation · Math Problems (w/ Reasoning) | ROC AUC: 99.96$\pm$0.21 -> 99.99$\pm$0.07 | Helped |
| add the proposed second-order loss-change term (ablation) | Influence estimation with standard parameter change · CIFAR-10N Worst mislabeled detection | ROC AUC: 41.52 -> 95.93 | Helped |
| add the proposed second-order loss-change term (ablation) | Influence estimation with proposed parameter change · CIFAR-10N Worst mislabeled detection | ROC AUC: 33.38 -> 96.46 | Helped |
| replace standard parameter change with the proposed parameter change (ablation) | Influence estimation with standard loss change · CIFAR-10N Worst mislabeled detection | ROC AUC: 41.52 -> 33.38 | Hurt |
| replace standard parameter change with the proposed parameter change (ablation) | Influence estimation with proposed loss change · CIFAR-10N Worst mislabeled detection | ROC AUC: 95.93 -> 96.46 | Helped |
| replace SAM with F-SAM (ablation) | FVM · CIFAR-10N Worst mislabeled detection | ROC AUC: 96.46$\pm$0.08 -> 96.76$\pm$0.10 | Helped |
| reduce validation set size from 10000 to 1000 (ablation) | FVM · CIFAR-10N Worst mislabeled detection | ROC AUC: 96.46 $\pm$ 0.08 -> 93.72 $\pm$ 0.53 | Hurt |
Surprising findings
- Standard influence estimates for clean samples approach zero during tuning toward flat validation minima, undermining the error bound.
- Replacing the diagonal Fisher preconditioner with LiSSA or DataInf does not improve results, and the diagonal approximation is often best.
Limitations
- Directly optimizing the sharpness objective is computationally expensive, so the method falls back on validation minima (VM) or SAM-based approximations.
- FVM underperforms some baselines in text and image generation, attributed to a simplified LPF setting (M=1 instead of 8).
- The theoretical bound depends on assumptions about mean influence being nonzero for each class and on a linear or second-order parameter-change approximation.
Experimental setup
Datasets: CIFAR-10N CIFAR-100N CIFAR-10 CIFAR-100 DreamBooth sentence transformations math word problems
Benchmarks: CIFAR-10N CIFAR-100N sentence transformations math word problems DreamBooth style generation DreamBooth subject generation
Metrics: ROC AUC AP top-1 relabeling accuracy Recall
Baselines: LiSSA TracIn GEX DataInf EK-FAC LPF-based variants
Builds on: influence functions (Koh and Liang) TracIn LiSSA DataInf Sharpness-Aware Minimization LPF
Closest papers by meaning
- Influence Functions in Deep Learning Are Fragile (ICLR 2021 · similarity 0.91)
- A Versatile Influence Function for Data Attribution with Non-Decomposable Loss (ICML 2025 · similarity 0.87)
- Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models (ICML 2025 · similarity 0.87)
- Rescaled Influence Functions: Accurate Data Attribution in High Dimension (NeurIPS 2025 · similarity 0.86)
- Influence Functions for Edge Edits in Non-Convex Graph Neural Networks (NeurIPS 2025 · similarity 0.85)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.