Research map / Large language models / Instruction and fine-tuning
The Best Instruction-Tuning Data are Those That Fit
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRAPE, which gathers multiple candidate responses per instruction and selects the one with the highest length-normalized probability under the target pretrained model before standard SFT, showing gains over larger-data and strongest-teacher baselines.
Problem
SFT data responses often come from sources out of the target model's pretrained distribution, causing diminishing returns and degraded performance and robustness when scaling synthetic or mixed SFT data.
Approach
For each instruction, pick the candidate response with the highest length-normalized log-probability under the target base model (lowest perplexity), then run standard SFT on these model-aligned responses; this requires only forward passes and no new generation or training.
Key results
- On UltraInteract with Qwen2.5-7B, GRAPE averages 56.7 vs 42.9 for Llama3.1-405B-distilled data (+13.8) and 39.4 for 3x data (+17.3).
- On the Tulu-Olmo combo with Llama3.1-8B, GRAPE averages 35.9 avg versus 26.9 for all available data (1.58M) and 32.4 for Tulu3-SFT, with AlpacaEval2 LC 14.8 vs 12.4 for Tulu3-SFT.
- Reward-based selection with Skywork-Reward underperforms GRAPE (e.g., 35.9 vs 34.9 avg on Llama3.1-8B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select responses with highest normalized probability instead of original UltraInteract responses | SFT on UltraInteract-SFT · UltraInteract-SFT | average performance: 32.7 -> 36.3; 36.8 -> 41.5; 20.3 -> 24.1; 45.1 -> 56.7 | Helped |
| Select target-model-aligned responses instead of responses from Llama3.1-405B | SFT on UltraInteract-SFT · UltraInteract-SFT | average performance: 32.7 -> 36.3; 38.8 -> 41.5; 18.9 -> 24.1; 42.9 -> 56.7 | Helped |
| Select target-model-aligned responses instead of training on 3x responses | SFT on UltraInteract-SFT · UltraInteract-SFT | average performance: 32.7 -> 36.3; 31.7 -> 41.5; 19.0 -> 24.1; 39.4 -> 56.7 | Helped |
| Select aligned responses instead of all available data | SFT on Tulu-Olmo collection · Tulu-Olmo collection | average performance: 33.2 -> 35.9; 26.9 -> 31.9; 42.4 -> 48.6 | Helped |
| Select aligned responses instead of S2L-selected data | SFT on Tulu-Olmo collection · Tulu-Olmo collection | average performance: 31.7 -> 35.9; 29.6 -> 31.9; 44.7 -> 48.6 | Helped |
| Use highest-perplexity responses rather than GRAPE-selected responses (ablation) | GRAPE response selection · Tulu-Olmo collection | average performance | Hurt |
| Use random responses instead of GRAPE-selected responses (ablation) | GRAPE response selection · Tulu-Olmo collection | average performance | Hurt |
| Use only the original SFT response instead of GRAPE-selected responses (ablation) | GRAPE response selection · Tulu-Olmo collection | average performance | Hurt |
| Replace diverse external supervision with self-distilled outputs (ablation) | SFT on UltraInteract-SFT · UltraInteract-SFT | average performance: 36.3 -> 28.4; 41.5 -> 29.4; 24.1 -> 15.1 | Hurt |
| Fine-tune on responses generated by the previously fine-tuned model (ablation) | Self-generation on MATH · MATH | accuracy: 18.2 -> 15.9; 26.2 -> 23.6; 22.3 -> 20.6; 28.1 -> 21.4 | Hurt |
| Select the closest response rather than a random response per instruction (ablation) | Mistral-7B SFT on UltraInteract-SFT · UltraInteract-SFT | average performance: 27.1 -> 29.8 | Helped |
| Select lowest-perplexity instead of highest-perplexity trajectories (ablation) | Qwen2.5 long-chain-of-thought SFT · MATH | accuracy: 0.330 -> 0.396; 0.524 -> 0.544 | Helped |
Surprising findings
- Self-distillation (training on responses from the fine-tuned model itself) degrades performance versus the original data, e.g., Mistral-7B avg 28.4 vs 32.7.
- Adding more responses (3x data or all available data) often does not improve and can hurt performance relative to selection.
Limitations
- Assumes a quality-controlled candidate pool; selection quality depends on the base model's own capabilities.
- Pursuing in-distribution answers naively can degrade performance (self-distillation results).
Experimental setup
Datasets: UltraInteract-SFT Tulu-3 SFT mixture OLMo-2 SFT data OpenHermes-2.5 Magpie-Zoo OpenR1-Math LUFFY OpenThoughts-114k
Benchmarks: HumanEval MBPP LeetCode MATH GSM-Plus TheoremQA AlpacaEval 2 BBH MMLU TruthfulQA
Metrics: pass@1 accuracy length-controlled win rate raw win rate
Baselines: Original-UI Llama3.1-405B distillation 3x Data Random Reverse-GRAPE Highest perplexity Tulu3-SFT All Responses All Available Data S2L Emb-NV LESS
Builds on: Tulu 3 OLMo 2 UltraInteract LESS S2L
Closest papers by meaning
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
- Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning (ICLR 2025 · similarity 0.90)
- #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models (ICLR 2024 · similarity 0.89)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.87)
- AlpaGasus: Training a Better Alpaca with Fewer Data (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.