Research map / Large language models / Instruction and fine-tuning
DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DataInf, a closed-form influence approximation (Sherman-Morrison plus swapping inverse and average) that is fast and memory-efficient for LoRA-tuned LLMs and diffusion models.
Problem
Computing influence functions for training data is computationally and memory expensive for large models such as LLMs and text-to-image diffusion models, especially due to inverse Hessian computation.
Approach
Swap the order of matrix inversion and averaging in the damped Gauss-Newton (gradient second-moment) matrix, then apply Sherman-Morrison to obtain a closed-form inverse for each layer; the resulting estimator costs O(nd) with O(d) memory and is argued to be accurate when LoRA parameter counts are small.
Key results
- On GLUE-MRPC with r=1, DataInf correlates 0.64 with exact influence vs 0.50 for Hessian-free and 0.45 for LiSSA.
- On GLUE-QQP, DataInf takes 13 s vs 70 s for LiSSA and 11279 s for Exact to compute 4500 influence values; on average 5.5x faster than LiSSA and 1149.6x faster than Exact.
- On text-to-image style generation with stable-diffusion-v1.5, DataInf improves class detection AUC from 0.692 (Hessian-free) to 0.820.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DataInf instead of Hessian-free for influence estimation | Llama-2-13B-chat · Sentence transformations | class detection AUC: 0.999 ± 0.002 -> 1.000 ± 0.000 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | Llama-2-13B-chat · Math problems (no reasoning) | class detection AUC: 0.770 ± 0.174 -> 1.000 ± 0.000 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | Llama-2-13B-chat · Math problems (with reasoning) | class detection AUC: 0.772 ± 0.173 -> 1.000 ± 0.001 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | stable-diffusion-v1.5 · Text-to-image style generation | class detection AUC: 0.692 ± 0.007 -> 0.820 ± 0.005 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | stable-diffusion-v1.5 · Text-to-image subject generation | class detection AUC: 0.820 ± 0.000 -> 0.865 ± 0.000 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | Llama-2-13B-chat · Sentence transformations | class detection recall: 0.985 ± 0.033 -> 0.997 ± 0.010 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | Llama-2-13B-chat · Math problems (no reasoning) | class detection recall: 0.258 ± 0.388 -> 0.999 ± 0.006 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | Llama-2-13B-chat · Math problems (with reasoning) | class detection recall: 0.258 ± 0.388 -> 0.996 ± 0.025 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | stable-diffusion-v1.5 · Text-to-image style generation | class detection recall: 0.533 ± 0.008 -> 0.687 ± 0.006 | Helped |
| Use DataInf instead of Hessian-free for influence estimation | stable-diffusion-v1.5 · Text-to-image subject generation | class detection recall: 0.210 ± 0.003 -> 0.315 ± 0.003 | Helped |
| LoRA fine-tuning | Llama-2-13B-chat · Sentence transformations | accuracy: 0.01 -> 0.35 | Helped |
| Use DataInf instead of LiSSA for influence computation | RoBERTa-large · GLUE-QQP (4500 influence values) | runtime: 70 seconds -> 13 seconds | Helped |
Surprising findings
- LiSSA was highly unstable and in influential-data experiments collapsed to Hessian-free even after gradient rescaling.
- DataInf sometimes outperformed the exact influence function on mislabeled data detection, which the authors attribute partly to the damping parameter degrading Exact.
Limitations
- Approximation error bound scales as O(d_l^2), so accuracy is argued only for small LoRA parameter counts; no guarantee for full fine-tuning.
- Exact influence is infeasible at LLM/diffusion scale, so approximation quality is validated only on RoBERTa-large; large-model evaluation relies on proxy metrics like class detection.
- Evaluation of influence utility for generative models uses proxies (same-class AUC, mislabeled loss increase) rather than established downstream metrics.
Experimental setup
Datasets: GLUE-MRPC GLUE-SST2 GLUE-WNLI GLUE-QQP Sentence transformations (synthetic) Math word problems (synthetic) cartoon image-text pairs pixel-art image-text pairs line sketch image-text pairs DreamBooth
Benchmarks: GLUE-MRPC GLUE-SST2 GLUE-WNLI GLUE-QQP DreamBooth Sentence transformations Math word problems
Metrics: Pearson correlation with exact influence AUC for mislabeled detection runtime class detection AUC class detection recall FID
Baselines: Exact influence LiSSA Hessian-free EK-FAC
Builds on: influence functions LiSSA EK-FAC LoRA
Closest papers by meaning
- Influence Functions for Scalable Data Attribution in Diffusion Models (ICLR 2025 · similarity 0.87)
- Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · similarity 0.86)
- Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data (NeurIPS 2025 · similarity 0.86)
- LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution (NeurIPS 2026 · similarity 0.86)
- DeepSN: A Sheaf Neural Framework for Influence Maximization (AAAI 2025 · similarity 0.85)
- A Versatile Influence Function for Data Attribution with Non-Decomposable Loss (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.