Research map / Large language models / Reasoning and chain of thought

What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LoGra, a Kronecker-structured low-rank gradient projection for influence functions, plus the Logix software package, enabling LLM-scale data valuation on a 1B-token dataset with Llama3-8B-Instruct, GPT2-XL and Pythia-1.4B.

Problem

Influence-function data valuation does not scale to LLMs and billion-token training sets because per-sample gradients and projection matrices are prohibitively large in compute and memory, and training gradients must be recomputed for each query.

Approach

Impose a Kronecker-product structure on the random projection so that projected gradients are computed directly from projected forward and backward activations (via small encoder/bottleneck/decoder add-on layers, LoRA-like), cutting projection cost from O(bnk) to O(b sqrt(nk)); store projected training gradients on disk for reuse; interpret the Hessian damping term as spectral gradient sparsificat

Key results

• LoGra achieves up to 6,500x higher influence-computation throughput and 5x lower GPU memory than EKFAC influence on Llama3-8B-Instruct with a 1B-token dataset (79003.9 vs 12.2 pairs/s; 15 GB vs 75 GB).
• LoGra slightly underperforms EKFAC influence but noticeably outperforms gradient dot product, TRAK, and representation similarity on counterfactual evaluations (FMNIST, CIFAR-10, WikiText).
• LoGra-PCA outperforms LoGra-random on FMNIST and CIFAR-10, but underperforms on WikiText+GPT2, so random initialization was used for LLM experiments.

Measured effects reported in the paper

Surprising findings

• Naive influence functions frequently return outlier data with high gradient norms as most valuable data, mitigated with l-RelatIF normalization.
• Qualitative alignment between top valuable data and LLM outputs is weaker for Pythia-1.4B, which the authors attribute to lower output quality and lower-quality training data.

Limitations

• Influence functions are susceptible to outlier data with large gradient norms, especially for language modeling; mitigated only with self-influence normalization.
• Valuation dataset limited to a 1B-token OpenWebText subset due to storage; failure cases with low-quality models like Pythia-1.4B.
• PCA initialization was ad hoc for Transformers (GPT2) and did not help there; no high-performance vector database support implemented.

Experimental setup

Datasets: OpenWebText FashionMNIST CIFAR-10 WikiText

Benchmarks: FashionMNIST brittleness and LDS CIFAR-10 brittleness and LDS WikiText GPT2 brittleness and LDS OpenWebText 1B-token valuation efficiency

Metrics: brittleness test misclassification fraction linear datamodeling score (Spearman) throughput (tokens/s, pairs/s) GPU memory qualitative similarity

Baselines: gradient dot product TRAK EKFAC influence representation similarity Arnoldi IF

Builds on: influence functions TRAK EKFAC influence KFAC LoRA Arnoldi influence functions

Closest papers by meaning

• Convex Dataset Valuation for Post-Training (ICML 2026 · similarity 0.87)
• KAIROS: Scalable Model-Agnostic Data Valuation (NeurIPS 2025 · similarity 0.85)
• GMValuator: Similarity-based Data Valuation for Generative Models (ICLR 2025 · similarity 0.84)
• Fairshare Data Pricing via Data Valuation for Large Language Models (NeurIPS 2025 · similarity 0.84)
• NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.83)
• A Theory of Training Profit-Optimal LLMs (NeurIPS 2026 · similarity 0.83)