Research map / Large language models / Instruction and fine-tuning
SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelectIT, which selects high-quality instruction-tuning data using an LLM's own token-, sentence-, and model-level uncertainty, and releases the Selective Alpaca dataset (top 20% of Alpaca-GPT4).
Problem
Instruction data selection for LLM instruction tuning usually relies on external closed-source models, extra fine-tuned models, or supplementary datasets, which raises cost and limits adoption.
Approach
Rate each instruction sample from 1 to K by the foundation model's own next-token score distribution, refine it with uncertainty across semantically similar rating prompts (sentence-level) and across LLMs of different sizes weighted by parameter count (model-level), without any extra resources.
Key results
- On LLaMA-2-13B, SelectIT over full Alpaca-GPT4 improves the overall average from 44.7 to 47.1 (+2.4) using the top 20% of data.
- Selective Alpaca improves Mistral-7B by +1.5 and LLaMA-3-8B by +1.6 overall average, without selection by those models.
- SelectIT-selected ALMA-13B improves COMET from 83.7 to 84.2 on WMT22 translation tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select top 20% of Alpaca-GPT4 with SelectIT instead of using the full Alpaca-GPT4 dataset | LLaMA-2-7B · open-instruct | AVG: 34.1 -> 36.2 | Helped |
| Select top 20% of Alpaca-GPT4 with SelectIT instead of using the full Alpaca-GPT4 dataset | LLaMA-2-13B · open-instruct | AVG: 44.7 -> 47.1 | Helped |
| Add Token-R to vanilla Alpaca-GPT4 tuning (ablation) | LLaMA-2-7B · open-instruct | AVG: 34.1 -> 34.5 | Helped |
| Add Sentence-R to vanilla Alpaca-GPT4 tuning (ablation) | LLaMA-2-7B · open-instruct | AVG: 34.1 -> 35.3 | Helped |
| Add Model-R to vanilla Alpaca-GPT4 tuning (ablation) | LLaMA-2-7B · open-instruct | AVG: 34.1 -> 35.1 | Helped |
| Add Token-R to vanilla Alpaca-GPT4 tuning (ablation) | LLaMA-2-13B · open-instruct | AVG: 44.7 -> 45.1 | Helped |
| Add Sentence-R to vanilla Alpaca-GPT4 tuning (ablation) | LLaMA-2-13B · open-instruct | AVG: 44.7 -> 45.9 | Helped |
| Add Model-R to vanilla Alpaca-GPT4 tuning (ablation) | LLaMA-2-13B · open-instruct | AVG: 44.7 -> 45.5 | Helped |
| Vary the SelectIT rating parameter K across 3, 5, 7, and 9 (ablation) | LLaMA-2-7B and LLaMA-2-13B · open-instruct | AVG: 40.5, 41.7, 41.5, 41.4 | No clear effect |
| Increase uncertainty weight alpha from 0.2 to 0.8 (ablation) | LLaMA-2-7B · open-instruct | AVG: 36.2 -> 34.4 | Hurt |
| Select 20% of Alpaca-GPT4 by sample length rather than with SelectIT (ablation) | LLaMA-2-7B and ALMA · open-instruct and machine translation | reported overall scores: 35.5, 47.1, 30.1, 31.8 vs 36.2, 47.1, 30.5, 32.2 | Mixed |
| Randomly select 20% from the full dataset or the unselected remainder (ablation) | LLaMA-2 and ALMA · Alpaca and machine translation | reported overall scores: 34.1, 45.1, 29.3, 31.0; -0.4 | Hurt |
Surprising findings
- Selected data is longer and contains a higher proportion of computational problems, and performance drops when computation-heavy data falls below about 6%.
- Sentence-level self-reflection contributes most to the final selection, more than token-level or model-level reflection.
Limitations
- Only models below 30B parameters were studied due to computational constraints.
- Evaluated mainly on the Alpaca dataset; the 20% selection threshold is tuned for Alpaca and may not transfer.
- Model-level weighting uses parameter count as a proxy for capability, which is a heuristic.
Experimental setup
Datasets: Alpaca-GPT4 WizardLM Orca-GPT4 WMT17-20 test sets Flores-200
Benchmarks: MMLU BBH GSM TydiQA HumanEval AlpacaEval WMT22
Metrics: accuracy pass@10 win rate BLEU COMET
Baselines: Alpaca-GPT4 LIMA AlpaGasus Q2Q Instruction Mining WizardLM Orca-GPT4
Builds on: AlpaGasus LIMA Instruction Mining Q2Q
Closest papers by meaning
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.90)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.89)
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024 · similarity 0.89)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.89)
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · similarity 0.88)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.