Research map / Learning theory and algorithms / Prediction and decision losses
Capturing the Temporal Dependence of Training Data Influence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes data value embedding, a dot-product-based approximation of trajectory-specific leave-one-out influence that captures data ordering in SGD training, and uses it to reveal three phases of data influence during LLM pretraining.
Problem
Influence functions assume permutation-invariant, converged training, so they cannot capture how the timing and ordering of training data affects its influence on foundation models trained with stochastic, non-convergent, multi-stage schedules.
Approach
Define trajectory-specific LOO (removing a point from its specific training iteration) and approximate it via a backward-recursive data value embedding under GGN Hessian approximation, so influence on any test point is a dot product between the test gradient and the stored embedding, with influence checkpointing for parallel computation.
Key results
- Data value embedding achieves high Spearman correlation with ground-truth LOO on MNIST, while influence function correlates weakly across single- and all-epoch removal settings.
- Computing data value embeddings is over 15x faster than LoGRA (throughput 667.52 vs 41.6 data points/s) and uses 0.84GB peak GPU memory vs 63.6GB for LoGRA on Pythia-410M.
- Online data selection only in the first 2000 and after 20000 iterations closely matches full-run selection performance on Pile while reducing computation by more than 5x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Data Value Embedding instead of LoGRA to store influence vectors | Pythia-410M on 1% of Pile · Pile | storage throughput: 41.6 -> 667.52 | Helped |
| Use Data Value Embedding instead of LoGRA to store influence vectors | Pythia-410M on 1% of Pile · Pile | peak GPU memory for storing influence vectors: 63.6GB vs 0.84GB | Helped |
| Use Data Value Embedding instead of LoGRA to store influence vectors | Pythia-410M on 1% of Pile · Pile | storage: 170GB vs 171GB | No clear effect |
| Use Data Value Embedding instead of LoGRA for influence computation | Pythia-410M on 1% of Pile · Pile | influence computation throughput: 640 vs 640 | No clear effect |
| Apply online data selection only in the first 2000 iterations and after 20000 iterations rather than throughout training | Pythia-410M training on Pile · Pile | performance | No clear effect |
| Apply online data selection only in the first 2000 iterations and after 20000 iterations rather than throughout training | Pythia-410M training on Pile · Pile | computational cost: more than 5 times | Helped |
| Apply online data selection only in the first very brief high-influence region | Pythia-410M training on Pile · Pile | performance gain: less than 4%; ≈ 50% | Helped |
| Use Data Value Embedding rather than influence functions to estimate LOO | MLP trained on MNIST for 3 epochs · MNIST single epoch removal and all-epoch removal | Spearman correlation with ground-truth LOO | Helped |
| Use data value embeddings to estimate data influence across training stages | Pythia-410M · Pile validation set | data influence | Mixed |
Surprising findings
- Data points from the very early warmup phase retain higher influence on the final model than many later points, despite large influence drops during training.
- Influence function assigns identical scores to duplicate points regardless of training position and may rank irrelevant data highly (e.g., popular music for a military video game query).
Limitations
- Derived specifically for SGD; not directly extendable to Adam due to normalization terms, so SGD is used as a proxy.
- Storage cost of embeddings is substantial (O(BT p~)).
- Error bound relies on strong assumptions and no multiplicative guarantee is provided.
Experimental setup
Datasets: Pile Wikitext-103 MNIST
Benchmarks: MNIST
Metrics: Spearman correlation with ground-truth LOO GPU memory throughput storage validation loss
Baselines: influence function LoGRA gradient cosine similarity batch selection
Builds on: influence functions LoGRA TracIN In-Run Data Shapley
Closest papers by meaning
- A Versatile Influence Function for Data Attribution with Non-Decomposable Loss (ICML 2025 · similarity 0.87)
- The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes (CVPR 2024 · similarity 0.86)
- Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining (NeurIPS 2025 · similarity 0.86)
- Influence Functions for Scalable Data Attribution in Diffusion Models (ICLR 2025 · similarity 0.86)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.85)
- Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.