Research map / Architectures and efficiency / Mixture of experts
Grouter: Decoupling Routing from Representation for Accelerated MoE Training
Yuqi Xu, Rizhen Hu, Zihan Liu, Mou Sun, Kun Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add positional information to the MHA Transformer encoder Grouter (ablation) | Grouter · Mini-DS-V2-Lite, 10B tokens | validation loss: 3.367 -> 3.318 | Improved |
| use MHA Transformer encoder instead of decoder (ablation) | Grouter · Mini-DS-V2-Lite, 10B tokens | validation loss: 3.342 -> 3.318 | Improved |
| use MHA instead of MLA in the encoder Grouter (ablation) | Grouter · Mini-DS-V2-Lite, 10B tokens | validation loss: 3.356 -> 3.318 | Improved |
| use Transformer encoder instead of residual linear layer (ablation) | Grouter · Mini-DS-V2-Lite, 10B tokens | validation loss: 3.375 -> 3.318 | Improved |
| add positional information to the residual linear Grouter (ablation) | Grouter · Mini-DS-V2-Lite, 10B tokens | validation loss: 3.359 -> 3.375 | Hurt |
| add positional information to the MLA decoder Grouter (ablation) | Grouter · Mini-DS-V2-Lite, 10B tokens | validation loss: 3.387 -> 3.335 | Improved |
| add positional information to the MLA encoder Grouter (ablation) | Grouter · Mini-DS-V2-Lite, 10B tokens | validation loss: 3.360 -> 3.356 | No clear effect |
| use Grouter routing prior instead of Z-Loss + Aux Loss | MoE pretraining · Mini-Qwen3 (50B tokens) | validation loss: 2.740 -> 1.763 | Improved |
| use Grouter routing prior instead of Z-Loss + Aux Loss | MoE pretraining · Mini-GPT-OSS (30B tokens) | validation loss: 3.346 -> 2.940 | Improved |
| use Grouter routing prior instead of Z-Loss + Aux Loss | MoE pretraining · Downstream tasks (6 benchmarks) | average score: average improvement of 2.80 across six benchmarks | Improved |
| use Grouter with expert-placement-based communication reduction on top of DeepEP | MoE training throughput · 1, 2 and 4 nodes | throughput speedup: 33.5%, 19.3%, and 15.5% on 1, 2, and 4 nodes | Improved |
| add lightweight load-balancing expert tuning (50M tokens) to Grouter Raw (ablation) | Grouter · MoE pretraining (30B tokens) | validation loss / load balance | Improved |
Closest papers by meaning
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.89)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.88)
- Tight Clusters Make Specialized Experts (ICLR 2025 · similarity 0.88)
- Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization (ICLR 2025 · similarity 0.87)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.87)
- Ada-K Routing: Boosting the Efficiency of MoE-based LLMs (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
