Research map / Large language models / Language, safety and interpretability
MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Muffin, an instruction-tuning dataset built by diversifying multiple tasks per input text (facet-based brainstorming, instruction rematching, classification expansion), and shows T5-3B/11B and Llama2-13B tuned on it outperform prior synthetic datasets on four zero-shot benchmarks.
Problem
Instruction-tuning datasets are typically built by scaling inputs per instruction or by scaling input-free tasks; both yield models that either over-sensitize to inputs or generalize poorly to tasks requiring inputs.
Approach
Scale tasks per input: for each input text (from SuperNI and Dolma), generate many diverse instructions using input-facet-guided brainstorming and rematch human instructions to the input, then annotate outputs and expand classification tasks by generating wrong answer options.
Key results
- T5-11B trained on Muffin averages 44.58 across four benchmarks versus 36.55 for the best LLM-generated baseline (Self-Instruct); Muffin wins 3 of 4 benchmarks in direct comparison.
- In human evaluation, Muffin achieves 41.8 average acceptance ratio versus 34.4 for the strongest baseline (Self-Instruct).
- Muffin-trained data-quality check: 72.5% of 200 sampled instances correct on all three fields, versus 54% reported for a prior dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Muffin data instead of the strongest generated-data baseline | T5-3B · four zero-shot benchmarks; average score | average score: 4.42 average performance improvement | Helped |
| Use Muffin data instead of the strongest generated-data baseline | T5-11B · four zero-shot benchmarks; average score | average score: 8.03 average performance improvement | Helped |
| Add instruction rematching to instruction brainstorm (ablation) | T5-3B trained on 11.5k instructions · SuperNI validation | ROUGE-L (overall): 36.27 -> 37.53 | Helped |
| Add instruction rematching to instruction brainstorm (ablation) | T5-3B trained on 11.5k instructions · SuperNI validation | Exact-Match (classification): 28.26 -> 29.77 | Helped |
| Add classification expansion (ablation) | T5-3B trained on rematching plus brainstormed instructions · SuperNI validation | Exact-Match (classification): 29.77 -> 32.77 | Helped |
| Add classification expansion (ablation) | T5-3B trained on rematching plus brainstormed instructions · SuperNI validation | ROUGE-L (overall): 37.53 -> 41.20 | Helped |
| Use instruction rematching instead of instruction brainstorm (ablation) | T5-3B trained on 11.5k instructions · SuperNI validation | Exact-Match (classification): 28.26 -> 29.45 | Helped |
| Use instruction brainstorm instead of instruction rematching (ablation) | T5-3B trained on 11.5k instructions · SuperNI validation | ROUGE-L (generation): 46.11 -> 47.35 | Helped |
| Mix Muffin with SuperNI training data (ablation) | T5-3B · four zero-shot benchmarks; average performance | average performance | Hurt |
| Increase data size for Self-Instruct (ablation) | T5-3B trained on Self-Instruct · four evaluation benchmarks | average performance | Hurt |
| Increase data size for Dynosaur (ablation) | T5-3B trained on Dynosaur · four evaluation benchmarks | average performance | Hurt |
| Apply instruction brainstorming to inputs from Dynosaur or Unnatural Instruct (ablation) | T5-3B; Muffin-Dynosaur and Muffin-Unnatural · SuperNI-Test overall | ROUGE-L (overall): 32.55 -> 38.23; 37.12 -> 38.86 | Helped |
Surprising findings
- Mixing Muffin with human-crafted SuperNI data lowered performance below using Muffin alone, suggesting dataset paradigm matters more than data quality.
Limitations
- Relies on ChatGPT/GPT-4 for data generation, so API cost and model dependence; total cost about $571.
- Only 1,463 input texts; gains tied to the input set and to T5/Llama2 backbones.
- Some baselines are not trained at matched settings; human eval uses only 200 instances per benchmark and 5 volunteers.
Experimental setup
Datasets: Super-NaturalInstructions Dolma
Benchmarks: SuperNI-Test MMLU T0-Eval BBH
Metrics: Exact-Match ROUGE-L Rank classification accuracy human acceptance ratio pairwise preference
Baselines: Dolly LongForm Alpaca Alpaca-GPT4 WizardLM Self-Instruct Unnatural Instructions Dynosaur SuperNI T0 T0++ Flan-T5
Builds on: Super-NaturalInstructions Self-Instruct Unnatural Instructions Dynosaur Tk-Instruct
Closest papers by meaning
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.89)
- Instruction-Following Pruning for Large Language Models (ICML 2025 · similarity 0.88)
- DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation (AAAI 2024 · similarity 0.88)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.88)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
- Chain-of-Instructions: Compositional Instruction Tuning on Large Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.