Research map / Architectures and efficiency / Mixture of experts
Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynMoE, which uses a top-any sigmoid-threshold gating with per-expert thresholds and an adaptive process that adds and removes experts during training, removing the need to tune the number of experts and top-k.
Problem
Sparse Mixture of Experts performance depends heavily on hyper-parameters (number of experts K and top-k), making tuning costly; existing fixes either need extra hyperparameters (top-p) or dense warm-up.
Approach
Treat gating as multi-label classification: each token independently activates any subset of experts via cosine similarity against learnable per-expert thresholds (sign function with straight-through gradient), while experts are added when tokens activate none and removed when unused; a diversity-and-simplicity auxiliary loss encourages sparsity.
Key results
- On the vision-language setup with StableLM-1.6B, DynMoE-LLaVA (avg k=1.25, 1.75B activated) matches or exceeds MoE-LLaVA (k=2, 2.06B) on most benchmarks, e.g., MMB 63.2 vs 60.2 and VizWiz 40.6 vs 36.2.
- On GLUE with BERT-large, DynMoE averages 81.64 across COLA, MRPC, QNLI, MNLI, and RTE, within the range of the best fixed MoE settings (81.26 to 81.77).
- On ImageNet pretraining with ViT-S, DynMoE-B (k=1.25) reaches 68.5% Acc@1 versus 67.3% for DeepSpeed-MoE top-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic expert count and top-any gating instead of fixed top-k MoE | BERT-large MoE · GLUE average | score: 81.77 -> 81.64 | No clear effect |
| Use dynamic expert count and top-any gating instead of fixed top-2 routing | MoE-LLaVA with StableLM-1.6B · VQA-v2 | score: 76.7 -> 77.4 | Helped |
| Use dynamic expert count and top-any gating instead of fixed top-2 routing | MoE-LLaVA with Qwen-1.8B · VQA-v2 | score: 76.2 -> 76.4 | No clear effect |
| Use dynamic expert count and top-any gating instead of fixed top-2 routing | MoE-LLaVA with Phi-2-2.7B · VQA-v2 | score: 77.6 -> 77.9 | Helped |
| Add load-balance loss (ablation) | DynMoE with StableLM-1.6B · VQA-v2 | score: 77.4 -> 77.1 | Hurt |
| Add efficiency loss on top of load-balance loss (ablation) | DynMoE with StableLM-1.6B · VQA-v2 | score: 77.1 -> 77.1 | No clear effect |
| Weight expert-output averaging by gating scores (ablation) | DynMoE-LLaVA with StableLM-1.6B · VQA-v2 | score: 77.4 -> 73.9 | Hurt |
| Weight expert-output averaging by gating scores (ablation) | DynMoE-LLaVA with StableLM-1.6B · MME | score: 1300.9 -> 1176.8 | Hurt |
| Initialize new experts by averaging existing expert parameters (ablation) | DynMoE with ViT-S · TinyImageNet fine-tuning, E20 | accuracy: 86.70 -> 86.75 | Helped |
| Initialize new experts by weighted averaging existing expert parameters (ablation) | DynMoE with ViT-S · CIFAR-10 pretraining, epoch 500 | accuracy: 83.57 -> 82.01 | Hurt |
| Initialize new experts from the most frequently activated expert (ablation) | DynMoE with ViT-S · CIFAR-10 pretraining, epoch 500 | accuracy: 83.57 -> 80.58 | Hurt |
Surprising findings
- Weighting expert outputs by gating scores instead of plain averaging drops VQAv2 from 77.4 to 73.9 and MMBench from 63.2 to 52.1.
- In the vision-language experiments, the trained DynMoE model with Qwen-1.8B is roughly on par with MoE-LLaVA rather than clearly better.
Limitations
- The adaptive process and top-any gating are not efficient in the current implementation (routing overhead, no CUDA kernels).
- Gains over well-tuned MoE baselines are often within random fluctuation; some vision and language results are comparable rather than superior.
- Maximum number of experts is capped (16 for vision and language, 4 for vision-language), and all experts occupy GPU memory.
Experimental setup
Datasets: GLUE DomainBed PACS VLCS OfficeHome DomainNet ImageNet CIFAR-10 Tiny-ImageNet LLaVA-Finetuning
Benchmarks: GLUE DomainBed VQA-v2 GQA VisWiz ScienceQA-IMG TextVQA POPE MME MMBench LLaVA-Bench (in-the-Wild) MM-Vet
Metrics: accuracy activated parameters throughput latency FLOPs memory
Baselines: GMoE MoE-LLaVA top-p gating DeepSpeed-MoE LLaVA-1.5
Builds on: GMoE MoE-LLaVA MoEfication GShard auxiliary loss
Closest papers by meaning
- FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion (NeurIPS 2024 · similarity 0.87)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.86)
- Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion (NeurIPS 2024 · similarity 0.86)
- Monet: Mixture of Monosemantic Experts for Transformers (ICLR 2025 · similarity 0.85)
- Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models (NeurIPS 2026 · similarity 0.85)
- FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.