Research map / Large language models / Instruction and fine-tuning
LESS: Selecting Influential Data for Targeted Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LESS, an Adam-aware, gradient-datastore-based influence method that selects 5% of instruction data for targeted capabilities, with small models able to select data for larger or different-family models.
Problem
Selecting the most relevant instruction-tuning data from large mixed datasets to develop specific capabilities (targeted instruction tuning), given only a few target examples.
Approach
Adapts first-order gradient influence to the Adam optimizer using warmup LoRA training checkpoints, uses cosine similarity to avoid short-sequence bias, and projects LoRA gradients to low dimension to build a reusable gradient datastore for scoring training points against few-shot validation subtasks.
Key results
- LESS selecting 5% of data improves Llama-2-7B average over random 5% selection (49.3 vs 46.0 for the averaged table; MMLU 50.2 vs 46.5, TydiQA 56.2 vs 52.7, BBH 41.5 vs 38.9), and on TydiQA with Llama-2-7B it exceeds full-data training (56.2 vs 54.0).
- On Mistral-7B, LESS-T (selected with Llama-2-7B gradients) reaches TydiQA 61.7 vs full-data 57.7, showing cross-model and cross-family transfer.
- Baselines BM25, DSIR, and RDS give only small gains over random selection, while LESS beats the strongest baseline by 1.7-3.5 points on Llama-2-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select 5% using LESS instead of random selection | Llama-2-7B · MMLU | accuracy: 46.5 -> 50.2 | Helped |
| select 5% using LESS instead of random selection | Llama-2-7B · TydiQA | macro-averaged F1: 52.7 -> 56.2 | Helped |
| select 5% using LESS instead of random selection | Llama-2-7B · BBH | exact match: 38.9 -> 41.5 | Helped |
| select 5% using LESS instead of random selection | Llama-2-13B · MMLU | accuracy: 53.4 -> 54.0 | Helped |
| select 5% using LESS instead of random selection | Llama-2-13B · TydiQA | macro-averaged F1: 53.0 -> 54.6 | Helped |
| select 5% using LESS instead of random selection | Llama-2-13B · BBH | exact match: 47.0 -> 50.6 | Helped |
| select 5% using LESS instead of random selection | Mistral-7B · MMLU | accuracy: 60.0 -> 61.8 | Helped |
| select 5% using LESS instead of random selection | Mistral-7B · TydiQA | macro-averaged F1: 56.9 -> 60.3 | Helped |
| select 5% using LESS instead of random selection | Mistral-7B · BBH | exact match: 54.5 -> 56.0 | Helped |
| use LESS-selected 5% rather than the full dataset | Llama-2-7B · TydiQA | macro-averaged F1: 54.0 -> 56.2 | Helped |
| use LESS-selected 5% rather than the full dataset | Llama-2-7B · MMLU | accuracy: 51.6 -> 50.2 | Hurt |
| use Llama-2-7B gradient features to select 5% | Llama-2-13B · TydiQA | macro-averaged F1: 53.0 -> 57.5 | Helped |
Surprising findings
- Using pre-trained (no warmup) Llama-2-7B or Llama-2-7B-Chat gradients for selection significantly hurts LESS performance, showing warmup training is crucial.
- Training on only the few-shot validation data rarely helps over the base model, and instruction tuning on the 5% subset can slightly underperform the base model on MMLU for stronger models.
Limitations
- Requires a warmup LoRA training phase and expensive gradient feature computation before selection.
- Averaging gradients over completion tokens causes a sequence-length bias and may be less effective for long-form generation.
- First-order linear approximation ignores interactions between selected datapoints; minimizing validation loss does not always improve task accuracy.
Experimental setup
Datasets: Flan V2 CoT Dolly Open Assistant 1 MMLU TydiQA BBH GSM8K TruthfulQA
Benchmarks: MMLU TydiQA BBH GSM8K TruthfulQA
Metrics: 5-shot accuracy 1-shot macro F1 3-shot exact match exact match MC2
Baselines: random selection BM25 DSIR RDS full dataset training SGD influence SignGD influence
Builds on: influence functions TracIn LoRA Johnson-Lindenstrauss random projection DSIR
Closest papers by meaning
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.92)
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024 · similarity 0.90)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.90)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.90)
- SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection (NeurIPS 2024 · similarity 0.90)
- Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.