Research map / Architectures and efficiency / Mixture of experts
Expert Divergence Learning for MoE-based Language Models
Jiaang Li, Haibin Chen, Langming Liu, Yujin Yuan, Yadao Wang, Yizhen Zhang, Chengting Yu, Xin Tong, Weidong Zhang, Shilei Liu, Wenbo Su, Bo Zheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 49-class expert divergence loss | MoE 15B-A1.5B · 7-benchmark average | average accuracy: 35.59 -> 36.65 | Improved |
| add 3-class expert divergence loss | MoE 15B-A1.5B · 7-benchmark average | average accuracy: 35.59 -> 36.34 | Improved |
| add 49-class expert divergence loss | MoE 8B-A0.8B · 7-benchmark average | average accuracy: 35.10 -> 35.77 | Improved |
| add 3-class expert divergence loss | MoE 8B-A0.8B · 7-benchmark average | average accuracy: 35.10 -> 35.54 | Improved |
| add 49-class expert divergence loss | MoE 3B-A0.3B · 7-benchmark average | average accuracy: 34.13 -> 34.91 | Improved |
| add 3-class expert divergence loss | MoE 3B-A0.3B · 7-benchmark average | average accuracy: 34.13 -> 33.98 | No clear effect |
| use 49-class instead of 3-class domain labels (ablation) | expert divergence loss · 15B-A1.5B average | average accuracy: 36.34 -> 36.65 | Improved |
| use 49-class instead of 3-class domain labels (ablation) | expert divergence loss · 3B-A0.3B average | average accuracy: 33.98 -> 34.91 | Improved |
| treat each document as its own domain instead of 49 classes (ablation) | expert divergence loss · 3B training loss | LM loss | Hurt |
| add expert divergence loss | MoE 15B-A1.5B · Training throughput | k tokens/s/gpu: 5.07 -> 4.96 | No clear effect |
| add expert divergence loss | MoE 15B-A1.5B · Inference throughput (en_val_1000) | tokens/s: 14.45 -> 14.39 | No clear effect |
| vary divergence coefficient beta | expert divergence loss · 3B-A0.3B training loss | LM loss | No clear effect |
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.93)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.89)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.89)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
