Research map / Large language models / Instruction and fine-tuning
Influence-Preserving Proxies for Gradient-Based Data Selection in LLM FineTuning
Sirui Chen, Yunzhe Qi, Mengting Ai, Yifan Sun, Ruizhong Qiu, Jiaru Zou, Jingrui He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| build proxy with IProX instead of using off-the-shelf smaller model | Qwen3-4B target · Dolly, avg of MMLU/BBH/TyDiQA | average score: 63.81 (Qwen3-1.7B) -> 64.18 (IProX 1.5B) | Improved |
| build proxy with IProX (rho=0.3) instead of Llama3.2-1B off-the-shelf | Llama3.2-3B target · Dolly, avg of MMLU/BBH/TyDiQA | average score: 47.35 -> 48.97 | Improved |
| replace standard SVD with influence-preserving SVD (IPSVD) (ablation) | IProX · Llama3.2-3B, TyDiQA rho=0.7 | exact match: 32.73 -> 39.04 | Improved |
| add gradient-alignment stage (ablation) | IProX · Llama3.2-3B, TyDiQA rho=0.7 | exact match: 35.32 -> 39.79 | Improved |
| add KL anchoring of logits (ablation) | IProX · Llama3.2-3B, TyDiQA rho=0.7 | exact match: 37.66 -> 39.79 | Improved |
| use IProX instead of Layer Extraction and Influence Scorer baselines | Llama3.2-3B, Gemma3-4B · Dolly, avg | average score: +2.81 and +1.99 average over strongest baseline | Improved |
| use IProX with Influence Function instead of TracIn | Llama3.2-3B, Gemma3-4B · Dolly, avg | average score: 46.81 -> 49.56 (Llama3.2, rho=0.3) | Improved |
| increase proxy size (lower sparsity rho) (ablation) | IProX · four model families | average score: Llama3.2-3B: 47.54 (rho=0.7) -> 48.97 (rho=0.3) | Improved |
| increase probe set size beyond 3x default (ablation) | IProX · Llama3.2-3B rho=0.7 | average score: 47.85 (3x) vs 47.55 (5x) | No clear effect |
| reduce probe set diversity via 30% SMOTE redundancy (ablation) | IProX · Llama3.2-3B rho=0.7 | average score: 47.54 -> 46.48 | Hurt |
| set KL coefficient to 0.1 vs 0.5/0.01/0.001 (ablation) | IProX · Llama3.2-3B rho=0.7 | average score: 46.90/46.59/46.36 -> 47.54 | Improved |
| compress to extreme sparsity rho=0.9 | IProX vs Layer Extraction · Llama3.2-3B, Qwen2-7B | average score: 44.64 -> 46.67 (Llama3.2); 57.38 -> 59.64 (Qwen2-7B) | Improved |
Closest papers by meaning
- Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · similarity 0.89)
- BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.88)
- One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.88)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.87)
- Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data (NeurIPS 2025 · similarity 0.87)
- MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
