Research map / Architectures and efficiency / Mixture of experts
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Vectors (MoV) and Mixture of LoRA (MoLORA), extremely parameter-efficient MoEs that use (IA)^3 vectors or LoRA adapters as experts with soft merging, for instruction tuning T5 models on P3 and evaluating zero-shot on unseen tasks.
Problem
Conventional MoEs require storing all experts and are costly to fully fine-tune; the paper asks whether MoE can be applied within parameter-efficient fine-tuning for instruction tuning and generalization to unseen tasks.
Approach
Replace each MoE expert with a lightweight PEFT adapter ((IA)^3 vector or LoRA), train router and experts from scratch on a frozen pretrained model, and use soft merging (weighted average of experts before applying the linear adapter) so that only one merged adapter is applied per block.
Key results
- MoV-30 on T5-3B reaches 60.61 average median accuracy on unseen tasks, a 14.57% improvement over (IA)^3 (52.90) and slightly above fully fine-tuned T0-3B replication (60.06) with 0.68% of parameters.
- At 11B, MoV-60 (0.862% params) achieves 64.08 average median, versus 65.03 for the T0-11B replication and 59.12 for (IA)^3.
- Soft merging of 10 experts (59.93) outperforms top-1 (54.92) and top-2 (57.45) discrete routing for MoV-10 on T5-3B; token routing beats sentence-embedding routing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace single (IA)^3 with a 30-expert MoV | T5-3B instruction tuning · P3 unseen tasks | median accuracy (Average): 14.57% | Helped |
| Replace LORA with 15-expert MoLORA | T5-3B instruction tuning · P3 unseen tasks | median accuracy (Average): 5.70% | Helped |
| Increase MoV experts from 10 to 30 (ablation) | MoV on T5-3B · P3 unseen tasks | median accuracy (Average): 59.93 to 60.61 | Helped |
| Increase MoV experts from 30 to 60 (ablation) | MoV on T5-3B · P3 unseen tasks | median accuracy (Average): 60.61 to 59.83 | No clear effect |
| Increase MoV experts from 10 to 60 (ablation) | MoV on T5-770M · P3 unseen tasks | median accuracy (Average): 52.47 to 53.63 | Helped |
| Increase MoV experts from 10 to 60 (ablation) | MoV on T5-11B · P3 unseen tasks | median accuracy (Average): 62.3 to 64.08 | Helped |
| Increase MoLORA experts from 5 to 15 (ablation) | MoLORA on T5-3B · P3 unseen tasks | median accuracy (Average): 58.6, 59.77, and 60.79 | Helped |
| Use soft merging instead of discrete top-1 routing (ablation) | MoV-10 on T5-3B · P3 unseen tasks | median accuracy (Average): 54.92 -> 59.93 | Helped |
| Use soft merging instead of discrete top-2 routing (ablation) | MoV-10 on T5-3B · P3 unseen tasks | median accuracy (Average): 57.45 -> 59.93 | Helped |
| Add load-balancing loss to top-2 routing (ablation) | MoV on T5-3B · P3 unseen tasks | median accuracy (Average): 1.5 average median score | Hurt |
| Route using sentence embeddings instead of token embeddings (ablation) | MoV-10 on T5-3B · P3 unseen tasks | median accuracy (Average): 8.86% | Hurt |
| Use batch size 2048 instead of the small-batch training setup (ablation) | Parameter-efficient MoE · P3 unseen tasks | performance | Hurt |
Surprising findings
- Discrete top-k routing with load-balancing loss further decreased performance for k=2, and larger batch sizes caused experts to collapse to a single expert.
- Sentence-embedding routing (task-level inductive bias) performed worse than token routing across all model sizes.
Limitations
- Evaluated only on text-to-text T5 models; decoder-only models such as GPT-style not tested.
- Assessment limited to fine-tuning; pretraining efficacy not explored.
- MoE training sensitive to batch size and learning rate; MoLORA vs MoV ranking varies with model scale.
Experimental setup
Datasets: P3
Benchmarks: ANLI CB RTE WSC WIC Copa WinoGrande HellaSwag
Metrics: median accuracy mean accuracy
Baselines: T0 (full fine-tuning) (IA)^3 LoRA Polytropon
Builds on: (IA)^3 LoRA Mixture of Experts T0 Polytropon
Closest papers by meaning
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.93)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.91)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.