Research map / Architectures and efficiency / Mixture of experts
MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoDE, a mixture-of-experts training method where experts mutually distill (via L2 loss on expert outputs/embeddings) at moderate strength to counter "narrow vision," evaluated on tabular, translation, and image classification tasks.
Problem
Sparse or dense gated mixture-of-experts suffer from "narrow vision": each expert learns only from the limited samples routed to it, which limits generalization of the overall MoE.
Approach
Add a mutual distillation loss among experts (each expert distills from the average of the other experts' outputs or from peers via squared error on expert outputs) with a moderate strength so experts pick up features learned by others while keeping their specialization; also introduces "expert probing" to estimate per-expert performance on its dominating sub-task.
Key results
- On seven tabular datasets, MoDE with sparse gating improves over the SMoE baseline on all datasets (e.g., Isolet 0.9426 to 0.9546).
- On WMT14 En-De, MoDE reaches 29.48 BLEU vs 29.10 for MoE and 27.9 for the single model.
- On CIFAR-100, MoDE reaches 0.7824 accuracy vs 0.7545 for MoE and 0.7594 for the single model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add moderate mutual distillation among experts | DMoE and SMoE · Tabular benchmark datasets | accuracy: MoE -> MoDE is higher on all seven datasets for both gate types | Helped |
| Add mutual distillation among experts | Transformer MoE · IWSLT14, IWSLT17, WMT14 translation datasets | BLEU: MoE -> MoDE: 34.88 -> 35.14; 28.66 -> 28.91; 29.81 -> 33.17; 14.46 -> 15.40; 29.10 -> 29.48 | Helped |
| Add mutual distillation among experts | CNN MoE · MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100 | accuracy: MoE -> MoDE: 0.9886 -> 0.9918; 0.9057 -> 0.9083; 0.9444 -> 0.9519; 0.7545 -> 0.7824 | Helped |
| Increase expert count from 2 to 4 or 8 in MoDE (ablation) | MoDE · Tabular benchmark datasets | accuracy: MoDE accuracy is higher with 8 experts than 2 experts on all seven datasets | Helped |
| Increase distillation strength to α=100 (ablation) | MoDE · Mfeat-karhunen | overall accuracy: 0.9625 -> 0.9450 | Hurt |
Surprising findings
- Excessive distillation strength (alpha=100) makes experts converge to identical outputs (consistency 1.0) and lowers test accuracy, so distillation must be moderate.
- On IWSLT17 Ar-En, the base MoE (29.81 BLEU) underperforms the single model (32.92), while MoDE recovers to 33.17.
Limitations
- Experiments are limited to small tabular models and relatively small vision models; no large language model experiments (stated as future work).
- Expert probing is an approximation that masks the dominating expert, not the true MoE opinion.
- Distillation strength alpha must be tuned per domain (0.01-0.1 tabular, 1 NLP, 10 CV).
Experimental setup
Datasets: OpenML tabular datasets (Isolet, Mfeat-karhunen, Mfeat-factors, Optdigits, Satimage, First-order-theorem-proving, Artifi IWSLT14 En-De IWSLT17 En-Ar WMT14 En-De MNIST Fashion-MNIST CIFAR-10 CIFAR-100
Benchmarks: OpenML tabular datasets IWSLT14 En-De IWSLT17 En-Ar WMT14 En-De MNIST Fashion-MNIST CIFAR-10 CIFAR-100
Metrics: top-1 accuracy BLEU
Baselines: Single model MoE (DMoE) MoE (SMoE)
Builds on: Mixture of Experts (Jacobs et al. 1991) Knowledge distillation (Hinton et al. 2015) Deep mutual learning Sparsely-gated MoE (Shazeer et al. 2017)
Closest papers by meaning
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.90)
- Autonomy-of-Experts Models (ICML 2025 · similarity 0.90)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.89)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.