Research map / Probabilistic and scientific modeling / Molecules and proteins
The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Mirrored Influence Hypothesis (train-to-test influence correlates with test-to-train influence) and Forward-INF, which estimates training-data influence using only forward passes on training points plus continual-learning updates on a small test set.
Problem
Training data influence estimation is computationally expensive because existing methods require retraining on many subsets or computing gradients for every training point, which does not scale to large datasets and models.
Approach
Reformulates train-to-test influence as the reciprocal test-to-train influence (how training-point losses change when the model is updated on test samples), so that only the small test set needs backward passes and each training point needs just two forward passes.
Key results
- Forward-INF detects duplicated (leaked) samples with 100% top-1 accuracy on CIFAR-10 ResNet-18, versus 30% for Influence Functions and 0% for TracIn, while being 30-40x faster per test point.
- Mirrored influence correlation: Pearson 0.9975 (MNIST), 0.9939 (FMNIST), 0.9673 (CIFAR-10) in the near-noiseless logistic regression setting.
- On FTRACE-TREx behavior tracing, Forward-INF reaches MRR 0.2101 vs 0.1658 for TracIn (single checkpoint) at 15K candidates, with about 4x more inspected queries per minute (1306 vs 308).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Forward-INF instead of IF for leaked-example ranking | ResNet18 · CIFAR-10 data leakage detection | Top-1 accuracy: 30 -> 100 | Helped |
| Use Forward-INF instead of TracIn for leaked-example ranking | ResNet18 · CIFAR-10 data leakage detection | Top-1 accuracy: 2 -> 100 | Helped |
| Use Forward-INF instead of IF for leaked-example ranking | ResNet50 · CIFAR-100 data leakage detection | Top-1 accuracy: 0 -> 95 | Helped |
| Use Forward-INF instead of TracIn for leaked-example ranking | ResNet50 · CIFAR-100 data leakage detection | Top-1 accuracy: 10 -> 95 | Helped |
| Use Forward-INF instead of TracIn | MT5 · FTRACE-TREx, candidate set 15K | MRR: 0.1658 -> 0.2101 | Helped |
| Use Forward-INF instead of TracIn | MT5 · FTRACE-TREx, candidate set 20K | Precision: 0.1300 -> 0.1518 | Helped |
| Use Forward-INF instead of TracIn | MT5 · FTRACE-TREx | Queries inspected per minute: 307.522 -> 1306.323 | Helped |
| Use multiple checkpoints instead of a single checkpoint (ablation) | TracIn · FTRACE-TREx, candidate set 15K | MRR: 0.1658 -> 0.1596 | Hurt |
| Use multiple checkpoints instead of a single checkpoint (ablation) | TracIn · FTRACE-TREx, candidate set 20K | MRR: 0.1532 -> 0.1508 | Hurt |
| Use gradient ascent instead of gradient descent for test-set updating (ablation) | Forward-INF · Influence estimation | Influence estimation performance | No clear effect |
Surprising findings
- Using multiple TracIn checkpoints performed worse than a single checkpoint in behavior tracing.
- Gradient-based self-influence fails to detect randomly mislabeled samples because mislabeled points can have small gradient magnitudes, whereas Forward-INF's loss-difference score detects them.
Limitations
- Mirrored correlation is shown empirically; theory only covers single-point perturbations under convex or first-order approximations.
- Forward-INF uses continual learning on test samples, which can cause catastrophic forgetting and the paper does not add countermeasures.
- Group-to-group non-convex correlations are not theoretically explained.
Experimental setup
Datasets: MNIST FMNIST CIFAR-10 CIFAR-100 ImageNet100 LAION 2B FTRACE-TREx
Benchmarks: CIFAR-10 data leakage detection CIFAR-100 data leakage detection CIFAR-10 mislabeled data detection FTRACE-TREx behavior tracing
Metrics: Pearson correlation Spearman correlation top-K detection accuracy computation time MRR precision queries per minute
Baselines: Influence Functions TracIn Retraining-based memorization method (Feldman & Zhang) Information retrieval baseline
Builds on: Influence Functions TracIn Forward-INF
Closest papers by meaning
- Small-to-Large Generalization: Training Data Influences Models Consistently Across Scale (ICLR 2025 · similarity 0.86)
- Capturing the Temporal Dependence of Training Data Influence (ICLR 2025 · similarity 0.86)
- Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models (ICLR 2025 · similarity 0.86)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.86)
- Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data (NeurIPS 2025 · similarity 0.85)
- A Versatile Influence Function for Data Attribution with Non-Decomposable Loss (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.