Research map / Architectures and efficiency / Mixture of experts
Expert Merging in Sparse Mixture of Experts with Nash Bargaining
Dung Viet Nguyen, Anh Nguyen Thi, Minh Hoang Nguyen, Luc Nguyen, Shiqi Jiang, Ethan Fetaya, Linh Duy Tran, Gal Chechik, Tan Minh Nguyen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace EP-CAMEx's propagation with Nash bargaining weights (NAMEx) | EP-CAMEx · WikiText-103 small test | perplexity: 85.03 -> 84.12 | Improved |
| add Nash bargaining to EP-CAMEx | EP-CAMEx · ImageNet-1k Acc@1 | accuracy: 83.23 -> 84.06 | Improved |
| add Nash bargaining at ImageNet-A without momentum | EP-CAMEx · ImageNet-A | accuracy: 24.22 -> 25.32 | Improved |
| add complex momentum to expert propagation | NAMEx · ImageNet-A | accuracy: 25.32 -> 35.05 | Improved |
| add complex momentum | NAMEx-Full · WikiText-103 small test | perplexity: 83.16 -> 82.94 | Improved |
| use real momentum (phi=0) instead of complex momentum (ablation) | EP-CAMEx · GLUE RTE | accuracy: 76.23 -> 72.20 | Hurt |
| solve bargaining per-layer more often (delta l 1 or 5) vs once (ablation) | NAMEx · GLUE RTE | accuracy: 75.09 -> 77.98 (runtime 0.69 -> 1.14 s) | Improved |
| replace complex momentum with quaternion momentum (ablation) | NAMEx-Mom · GLUE MRPC | accuracy: 93.02 -> 93.24 | Improved |
| use mean instead of zero as disagreement point (ablation) | NAMEx-Full · MMLU/GSM8K/ARC DeepSeek-MoE | accuracy: deltas about 0.04 | No clear effect |
| increase bargaining iterations from 2 to 60 (ablation) | NAMEx-Full · Qwen1.5-MoE and DeepSeek-MoE | MMLU/GSM8K/ARC: e.g. MMLU 61.87 -> 61.81 | No clear effect |
| add NAMEx-Full to DeepSeek-MoE on GSM8K | DeepSeek-MoE · GSM8K zero-shot (Linear router) | accuracy: 16.53 -> 16.77 | Improved |
| add NAMEx to SMoE for CoLA/SST-2 classification variants | SMoE Top-2 · GLUE SST-2 | accuracy: 94.35 vs NAMEx 94.46 | Mixed |
Closest papers by meaning
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.87)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024 · similarity 0.86)
- Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making (NeurIPS 2025 · similarity 0.86)
- Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model (ICLR 2025 · similarity 0.85)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.85)
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
