Research map / Architectures and efficiency / Pruning and efficient tuning
Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a tiny 2-layer neural network (InfluenceNetwork, ~205K params) on a small subset of pairwise influence values from existing influence functions to predict the rest, cutting data valuation cost by 77-99% while matching downstream instruction-tuning subset selection performance.
Problem
Influence functions for instruction fine-tuning data selection require expensive forward/backward passes through large LMs and must be recomputed for new data, limiting scalability and generalization.
Approach
Distill the influence values of expensive LLM-based influence functions (DELIFT, DELIFT (SE), LESS, SelectIT) into a small learned InfluenceNetwork that takes concatenated bge-large-en-v1.5 embeddings of two data points and predicts their influence, so new points need no recomputation.
Key results
- InfluenceNetwork estimates DELIFT influence values with average MSE of 0.067 across quadrants using 5% of data pairs.
- NN-CIFT reduces data valuation time by 77-99% (e.g., DELIFT from ~68,000 s to ~215 s on Llama-8B) with average performance difference of 0.12-0.24% versus original influence functions in main tables.
- The InfluenceNetwork has 204,900 parameters, about 0.0007% of full LLM size as stated in the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate influence values with NN-CIFT instead of the original DELIFT influence function | DELIFT subset selection · Llama-8B across MixInstruct, Alpaca, and MMLU | performance across reported metrics: 0.13% | No clear effect |
| Estimate influence values with NN-CIFT instead of the original influence function | DELIFT, LESS, DELIFT (SE), and SelectIT · Qwen2.5 across MixInstruct, Alpaca, and MMLU | performance across reported metrics: 0.24% | No clear effect |
| Estimate influence values with NN-CIFT instead of the original influence function | DELIFT, LESS, DELIFT (SE), and SelectIT · Mistral across MixInstruct, Alpaca, and MMLU | performance across reported metrics: 0.12% | No clear effect |
| Use DistilGPT2 as the influence function's underlying language model instead of NN-CIFT | Influence functions for subset selection · Instruction fine-tuning results across evaluated models and datasets | performance | Hurt |
| Use NN-CIFT for valuation instead of the original influence functions | Data valuation · Reported valuation-cost experiments | valuation time: 77-99% faster | Helped |
| Replace the default embedding model with alternative embedding models (ablation) | InfluenceNetwork · DELIFT influence prediction, Q1–Q4 | MSE: 0.051; 0.084; 0.074; 0.084 -> 0.076; 0.087; 0.087; 0.089 | No clear effect |
| Increase InfluenceNetwork size by varying layer count and hidden dimensions (ablation) | InfluenceNetwork · Influence estimation | MSE | No clear effect |
| Increase the influence-network training fraction u (ablation) | InfluenceNetwork trained on DELIFT influence values · MixInstruct, DELIFT influence estimation | MSE | Helped |
| Increase the selected fine-tuning subset fraction v (ablation) | NN-CIFT subset-selection pipeline · MixInstruct and Alpaca | BGE performance | Mixed |
| Use larger IFT subsets for in-context learning (ablation) | NN-CIFT subset-selection pipeline · MixInstruct, ICL | BGE performance | Hurt |
| Replace the InfluenceNetwork with a random influence predictor (ablation) | Influence estimation · DELIFT influence estimation | MSE | Hurt |
| Replace the InfluenceNetwork with a zero-influence predictor (ablation) | Influence estimation · DELIFT influence estimation | MSE | Hurt |
Surprising findings
- Replacing the LLM in influence functions with DistilGPT2 (a small LM) substantially degrades downstream performance, especially for model-dependent methods, while NN-CIFT does not.
Limitations
- Relies on an external embedding model (bge-large-en-v1.5) whose cost is excluded from the reported efficiency numbers.
- Only studies data-to-data influence, not data-to-parameter influence; does not handle distributional shift (acknowledged as future work).
- Evaluated on a limited set of datasets (MixInstruct, Alpaca, MMLU) and single runs for performance tables.
Experimental setup
Datasets: MixInstruct Alpaca MMLU
Benchmarks: MixInstruct Alpaca MMLU
Metrics: ROUGE-1 BGE similarity LLM-as-judge score (Prometheus) accuracy MSE wall-clock time (seconds)
Baselines: DELIFT DELIFT (SE) LESS SelectIT DistilGPT2 Random Full Data
Builds on: DELIFT LESS SelectIT QLoRA
Closest papers by meaning
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.89)
- Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · similarity 0.89)
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations (NeurIPS 2025 · similarity 0.88)
- NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.87)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.87)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.