Research map / Large language models / Instruction and fine-tuning
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies complexity, quality, and diversity metrics for instruction-tuning data selection, proposes Evol Complexity/Quality scorers and an embedding-based diversity filter, and trains the Deita models on 6K selected samples.
Problem
Instruction tuning data is often heuristically built or manually selected, and there is no principled understanding of what makes good alignment data or how to select it automatically and efficiently.
Approach
Measure data along complexity, quality, and diversity; score complexity and quality with ChatGPT-ranked evolved variants used to train LLaMA scorers; then select samples greedily by combined score (c*q) subject to an embedding-distance diversity filter.
Key results
- Deita-Mistral-7B with 6K SFT samples reaches 7.22 MT-Bench and 80.78% AlpacaEval; the 10K variant reaches 7.32 MT-Bench, state of the art among open SFT-only 7B/13B models in the paper.
- Deita-Mistral-7B with 6K SFT + 10K DPO reaches 7.55 MT-Bench and 90.06% AlpacaEval.
- Deita uses about 10x less data (6K vs 125K for Vicuna-13B-v1.3) and matches or beats it on MT-Bench and Open LLM Leaderboard averages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select SFT data with Deita instead of random selection | LLaMA-1-13B · MT-Bench | score: 5.84 -> 6.46 | Helped |
| Increase Deita-selected SFT data from 6K to 10K examples (ablation) | LLaMA-1-13B · MT-Bench | score: 6.46 -> 6.60 | Helped |
| Increase Deita-selected SFT data from 6K to 10K examples (ablation) | Mistral-7B · MT-Bench | score: 7.22 -> 7.32 | Helped |
| Add DPO training with 10K preference pairs (ablation) | Deita-Mistral-7B 6K · MT-Bench | score: 7.22 -> 7.55 | Helped |
| Add DPO training with 10K preference pairs (ablation) | Deita-Mistral-7B 6K · Open LLM Leaderboard | average: 64.94 -> 69.86 | Helped |
| Select data using Evol Complexity instead of other complexity metrics | LLaMA-1-13B instruction tuning · X_sota and X_base | MT-Bench | Helped |
| Select data using Evol Quality instead of other quality metrics | LLaMA-1-13B instruction tuning · X_sota and X_base | MT-Bench | Helped |
| Use response length to select higher-quality data | LLaMA-1-13B instruction tuning · X_sota | MT-Bench | No clear effect |
| Select data with Repr Filter rather than random selection | LLaMA-1-13B instruction tuning · X_sota and X_base | MT-Bench | Helped |
| Use Repr Filter instead of Instag Diversity | LLaMA-1-13B instruction tuning · X_sota and X_base | MT-Bench | Helped |
| Select samples by response perplexity | LLaMA-1-13B instruction tuning · X_sota and X_base | MT-Bench | Hurt |
| Use model-based sentence representations rather than E5-Large-V2 semantic representations in Repr Filter (ablation) | Repr Filter · X_sota | MT-Bench | Helped |
Surprising findings
- Perplexity-based complexity selection performs much worse than random selection, as high-perplexity samples tend to have very short responses.
- Instruction length is not a good indicator of preferred alignment data.
Limitations
- Relies on ChatGPT (gpt-3.5-turbo) annotations for scorer training and on GPT-4 judging for MT-Bench, so results depend on proprietary models.
- Selection algorithm kept simple; more advanced selection methods left to future work.
- Direct Scoring and Instruction Node baselines were only run on a 50K subset due to annotation cost.
Experimental setup
Datasets: WizardLM (Alpaca) WizardLM (ShareGPT) UltraChat ShareGPT Alpaca Dolly OpenAssistant (OAssist) FLAN 2022 UltraFeedback LIMA test set
Benchmarks: MT-Bench AlpacaEval Open LLM Leaderboard (ARC, HellaSwag, MMLU, TruthfulQA)
Metrics: MT-Bench score AlpacaEval win rate (%) accuracy
Baselines: Random Selection Instag Complexity Instruction Node Direct Scoring IFD LIMA Alpagasus TAGLM Vicuna-13B-v1.3 WizardLM-13B Zephyr-beta Mistral-7B-Instruct-v0.2
Builds on: Evol-Instruct WizardLM InsTag Direct Preference Optimization Zephyr recipe
Closest papers by meaning
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.91)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.90)
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations (NeurIPS 2025 · similarity 0.89)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.89)
- SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection (NeurIPS 2024 · similarity 0.89)
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.