Research map / Large language models / Instruction and fine-tuning
Efficient Data Selection at Scale via Influence Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Influence Distillation, a second-order, landmark-based data selection framework that computes model-specific sample weights for LLM instruction tuning, matching or beating RDS+ and full-data training at up to 3.5x lower selection cost.
Problem
Efficient data selection for LLM fine-tuning toward a target distribution, where existing methods rely on heuristics, fixed representations, reference-model training, or expensive embeddings.
Approach
Derive a quadratic influence objective for sample weights (for both GD and Adam), solve it with non-negativity and sum constraints plus L2 regularization, and make it scalable by computing influence only on landmark samples and propagating to the rest via embedding-space linear coefficients using cheap JVP embeddings.
Key results
- On Llama2-7B, Influence Distillation averages +2.30 over Uniform, matching Full (+2.30) and beating RDS+ (+1.85) at 872 TF vs 2800 TF for RDS+.
- On Qwen2.5-1.5B, Influence Distillation reaches +6.52 average over Uniform vs +4.28 for RDS+ and +7.00 for Full, at 208 TF.
- Selection is reported as 2.9-3.5x faster than RDS+ in embedding+selection cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Influence Distillation instead of uniform sampling | Llama2-7B fine-tuning on Tulu V2 · Six target tasks, average | average improvement: +2.30 | Helped |
| Use Influence Distillation instead of uniform sampling | Llama3.2-3B fine-tuning on Tulu V2 · Six target tasks, average | average improvement: +0.77 | Helped |
| Use Influence Distillation instead of uniform sampling | Qwen2.5-1.5B fine-tuning on Tulu V2 · Six target tasks, average | average improvement: +6.52 | Helped |
| Use Influence Distillation instead of uniform sampling | Qwen2.5-3B fine-tuning on Tulu V2 · Six target tasks, average | average improvement: +0.18 | Helped |
| Use first-order Influence Distillation without the second-order term (ablation) | Influence Distillation · LLM instruction tuning | performance | No clear effect |
| Train on selected samples without incorporating their sample weights (ablation) | Influence Distillation · LLM instruction tuning | performance | No clear effect |
| Increase the number of landmarks (ablation) | Influence Distillation on Llama2-7B · Six target tasks | average improvement over Uniform | Helped |
| Use more than 2048 landmarks (ablation) | Influence Distillation on Llama2-7B · Six target tasks | average improvement over Uniform | Helped |
| Increase the pool size and number of selected samples (ablation) | Influence Distillation on Llama2-7B · MMLU | accuracy | Helped |
Surprising findings
- On Qwen2.5-3B, Influence Distillation (+0.18) and RDS+ (-0.28) show little gain over Uniform, and Full training gives 0.00 average, so gains are model-dependent.
- Applying the sample weights during training was found not to improve performance, so weights are used only for selection.
Limitations
- Requires a target dataset, so it does not directly apply to general data selection without a target distribution.
- Not validated for pre-training, where gradients shift over training and static selection may be insufficient.
- Warm-up phase cost is excluded from runtime measurements.
Experimental setup
Datasets: Tulu V2 MMLU GSM8k BBH TyDIQA Codex SQuAD CIFAR-10
Benchmarks: MMLU GSM8k BBH TyDIQA Codex SQuAD
Metrics: accuracy average improvement over Uniform TeraFLOPs for embedding+selection cost
Baselines: Uniform RDS+ Mid-PPL Full LESS
Builds on: LESS RDS+ Mid-PPL Adam SAM
Closest papers by meaning
- BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.91)
- Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data (NeurIPS 2025 · similarity 0.89)
- IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models (ICLR 2024 · similarity 0.89)
- MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models (NeurIPS 2024 · similarity 0.89)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.89)
- One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.