Research map / Large language models / Instruction and fine-tuning
Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that instruction tuning (Flan) reverses the weakness of sparse MoE models: Flan-MoE (up to 32B total params, Flan-ST-32B) outperforms dense Flan-PaLM 62B on held-out MMLU/BBH and other benchmarks at about one-third the FLOPs per token.
Problem
Sparse Mixture-of-Experts models often underperform dense models of equal compute when finetuned directly on downstream tasks, and it is unclear whether instruction tuning can make MoE models scale effectively for task-agnostic generalization.
Approach
Instruction-tune sparse MoE encoder-decoder models (Switch, GShard, expert-choice, ST-MoE) on the Flan collection; instruction tuning benefits MoE models more than dense models, so MoE + instruction tuning becomes superior to dense models at equal compute.
Key results
- Flan-ST-32B (32B total params, 32.1 GFLOPs/token) reaches 65.4% few-shot MMLU and 54.4% BBH, beating Flan-PaLM 62B (91.6 GFLOPs/token) on all four benchmark groups with normalized average 63.6 vs 57.6.
- Instruction tuning raises ST-32B normalized average by 45.2 points, versus 6.6 for PaLM 62B.
- Flan-MoE outperforms Flan-T5 at matched FLOPs across scales, with about 7.1% average MMLU-direct gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instruction-tune MoE models rather than leaving them without instruction tuning | MoE models · Normalized average across MMLU, BBH, reasoning, and QA | normalized average: +45.2% | Helped |
| increase active experts from top-1 token-choice to top-2 token-choice routing | Flan-MoE · MMLU Direct | accuracy: 38.0% to 39.9% | Helped |
| use expert-choice top-2 routing instead of GShard top-2 token-choice routing | Flan-MoE · MMLU and BBH across model scales | accuracy | Helped |
| freeze the gating function during instruction tuning (ablation) | Flan-EC-base · Average across MMLU, BBH, and GSM8K | average score: 37.7 -> 38.0 | Helped |
| freeze expert modules during instruction tuning (ablation) | Flan-EC-base · Average across MMLU, BBH, and GSM8K | average score: 37.7 -> 36.2 | Hurt |
| freeze MoE parameters during instruction tuning (ablation) | Flan-EC-base · Average across MMLU, BBH, and GSM8K | average score: 37.7 -> 36.2 | Hurt |
| use router Z-loss during instruction tuning (ablation) | Flan-EC-base · Average across MMLU, BBH, and GSM8K | average score: 37.7 -> 36.8 | Hurt |
| use balancing loss during instruction tuning (ablation) | Flan-EC-base · Average across MMLU, BBH, and GSM8K | average score: 37.7 -> 38.3 | Helped |
| freeze the gating function during instruction tuning (ablation) | Flan-ST-base · Average across MMLU, BBH, and GSM8K | average score: 37.8 -> 38.2 | Helped |
| freeze expert modules during instruction tuning (ablation) | Flan-ST-base · Average across MMLU, BBH, and GSM8K | average score: 37.8 -> 37.3 | Hurt |
| freeze MoE parameters during instruction tuning (ablation) | Flan-ST-base · Average across MMLU, BBH, and GSM8K | average score: 37.8 -> 36.9 | Hurt |
| use router Z-loss during instruction tuning (ablation) | Flan-ST-base · Average across MMLU, BBH, and GSM8K | average score: 37.8 -> 38.1 | Helped |
Surprising findings
- Without instruction tuning, MoE models underperform dense models of identical compute on downstream tasks.
- Flan-MoE models perform poorly on multilingual benchmarks (MGSM, TyDiQA), underperforming Flan-PaLM.
Limitations
- Poor multilingual performance (MGSM 15.5%, TyDiQA 25.1% for Flan-ST-32B) due to English-only instruction tuning.
- Still a considerable gap to Flan-PaLM 540B.
- MoE single-task finetuning prone to overfitting, with sensitivity to hyperparameters and auxiliary loss choice.
Experimental setup
Datasets: Flan collection (1,836 tasks: Muffin, T0-SF, NIV2, CoT) GLaM dataset (pretraining for MoE checkpoints)
Benchmarks: MMLU BBH GSM8K SVAMP ASDIV StrategyQA UnifiedQA AI2 science BoolQ ARC-easy ARC-challenge TyDiQA MGSM
Metrics: few-shot exact-match accuracy normalized average
Baselines: Flan-T5 Flan-PaLM PaLM T5 Flan-Switch Flan-GS Flan-EC
Builds on: Switch Transformer ST-MoE Flan instruction tuning GShard expert-choice routing
Closest papers by meaning
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.93)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.91)
- One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.