Research map / Architectures and efficiency / RNNs and normalization

LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Dataset distillation and data-efficient training · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LoRIF, which stores rank-c factors of projected per-example gradients and uses truncated SVD with the Woodbury identity to approximate the inverse Hessian, enabling larger projection dimensions for gradient-based training data attribution up to 70B models.

Problem

Projection-based influence-function methods for training data attribution face a quality-scalability tradeoff: storing projected per-example gradients (I/O-dominated query latency) and forming the D x D inverse Hessian (O(D^2) memory) both scale with projection dimension D, which quality needs to be

Approach

Store per-example projected gradients as rank-c factors (reducing storage/I/O from O(D) to O(c sqrt(D)) per layer) and approximate the damped Gauss-Newton inverse Hessian via a rank-r truncated SVD of the gradient matrix plus the Woodbury identity (reducing curvature memory from O(D^2) to O(Dr)).

Key results

• On GPT2-small, LoRIF (f=16, c=1, r=2^11) reaches LDS 0.1392 vs LoGRA 0.1370 at 4.1 GiB vs 35.9 GiB storage and 2.93 s vs 16.4 s latency.
• On OLMo-3-7B, LoRIF (f=16, c=1) achieves tail-patch 0.986 vs LoGRA 0.546 but with 647.4 GiB vs 1638.6 GiB storage; at f=128 LoRIF uses 20.3x less storage with tail-patch 0.495 vs 0.546.
• On Apertus-70B, LoRIF (f=64, c=1) improves tail-patch to 0.391 from LoGRA's 0.177 while using lower storage (1334.7 vs 1815.1 GiB) and latency (843.8 vs 1087.9 s).

Surprising findings

• Removing truncated SVD at c=1 with f=4 runs out of memory, while removing rank factorization restores the full storage cost (2297.7 GiB), confirming both components are needed.
• Rank-1 factorization on GPT2-small gives LDS 0.2073 at f=4, while rank-16 or rank-32 factorization at the same D gives lower LDS (0.3058 and 0.3428 vs 0.3876 without factorization), so higher c does not always improve quality.

Limitations

• Constructing the attribution index still requires a full gradient pass over the corpus, which is costly (e.g., ~68-200 hr preprocessing for 7B-70B models).
• Large-scale experiments cover supervised fine-tuning data, not full pre-training corpora.
• Large-scale evaluation uses tail-patch score with hyperparameters manually tuned, since LDS is infeasible at scale; GPT2-small LDS is the only retraining-based metric.

Experimental setup

Datasets: WikiText-103 Dolci Think SFT Apertus SFT data Nemotron Cascade SFT

Benchmarks: WikiText-103 LDS (GPT2-small) tail-patch score (OLMo-3-7B, Apertus-70B)

Metrics: LDS tail-patch score storage (GiB) query latency (s) avg relevance (1-5) preference rate

Baselines: LoGRA TrackStar GradDot EK-FAC RepSim

Builds on: LoGRA TRAK TrackStar EK-FAC LDS

Closest papers by meaning

• Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods (NeurIPS 2025 · similarity 0.89)
• A Versatile Influence Function for Data Attribution with Non-Decomposable Loss (ICML 2025 · similarity 0.88)
• GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection (NeurIPS 2025 · similarity 0.87)
• Influence Functions for Scalable Data Attribution in Diffusion Models (ICLR 2025 · similarity 0.87)
• Distributional Training Data Attribution: What do Influence Functions Sample? (NeurIPS 2025 · similarity 0.87)
• Enhancing Training Data Attribution with Representational Optimization (NeurIPS 2025 · similarity 0.87)