Research map / Large language models / Instruction and fine-tuning

Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Instruction data selection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Instruct-SkillMix, a two-stage automated pipeline that extracts instruction-following skills by prompting a strong LLM and then generates SFT data from random skill pairs, showing vanilla SFT on 4K examples yields strong AlpacaEval 2.0 and WildBench results.

Problem

Instruction tuning with open synthetic or crowd-sourced SFT datasets often fails to match proprietary instruct models; creating high-quality, diverse instruction data is costly and inconsistent.

Approach

Use a frontier LLM to extract a skill list by direct prompting (inspired by LLM metacognition), then generate Q&A pairs that combine randomly sampled k-tuples of skills plus a query type, yielding diverse, high-quality data for under $600.

Key results

• LLaMA-3-8B SFT on 4K Instruct-SkillMix examples reaches 42.76% LC win rate on AlpacaEval 2.0, above LLaMA-3.1-405B-Instruct (39.30%) and Claude 3 Opus (40.50%).
• Mistral-7B SFT on 4K examples achieves -29.25 WB-Reward on WildBench, outperforming Mistral Large and Claude 3 Sonnet.
• Replacing 20% of Instruct-SkillMix (k=2) 2K data with junk responses drops Mistral-7B LC win rate from 36.18% to 24.60% for the SkillMix-A variant, and from 31.57% to 0.77% for the SkillMix-D variant.

Measured effects reported in the paper

Surprising findings

• Replacing only 20% of training responses with junk data causes near-total collapse (0.77% LC win rate) for the skill-from-dataset variant, a super-proportionate harm.
• Minimizing validation cross-entropy loss selects a checkpoint with much lower downstream AlpacaEval performance (16.5% vs 29.77% LC WR), so generation-quality checkpoint selection is needed.

Limitations

• Performance saturates at about 4K examples, with held-out win rate near 50% against GPT-4.
• Pipeline does not yet cover under-specified or ambiguous queries or long-form generation, and does not address safety or alignment.
• Evaluations (AlpacaEval, WildBench, MT-Bench) use GPT-4-family judges that share family with the GPT-4-Turbo teacher, and have known blind spots.

Experimental setup

Datasets: Alpaca-52K UltraChat ShareGPT Instruct-SkillMix synthetic Q&A

Benchmarks: AlpacaEval 2.0 MT-Bench WildBench MMLU TruthfulQA GSM8K ARC Challenge Winogrande PIQA

Metrics: length-controlled win rate win rate MT-Bench score WB-Reward

Baselines: Alpaca-52K Alpaca-1K Longest ShareGPT UltraChat LLaMA-3-8B-Instruct Mistral-7B-Instruct-v0.2 Gemma-2-9B-Instruct Claude 3 Opus LLaMA-3.1-405B-Instruct GPT-4

Builds on: SkillMix LLM metacognition Self-Instruct Alpaca

Closest papers by meaning

• Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.91)
• The Best Instruction-Tuning Data are Those That Fit (NeurIPS 2025 · similarity 0.90)
• Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.89)
• MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.89)
• LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.89)
• AlpaGasus: Training a Better Alpaca with Fewer Data (ICLR 2024 · similarity 0.89)