Research map / Architectures and efficiency / Mixture of experts
Hybrid Routing for a Mixture of LoRA Experts
Yitong Huang, Ziqi Yang, Zihui Wang, Jianzhong Qi, Rongshan Yu, Xiaoliang Fan, Cheng Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard top-2 router with hybrid routing (expert-expert lower, token-expert higher) (ablation) | HotMoE · Task conflict (ARC-C/RTE/BoolQ/CommonGen/WMT19) | MRD: -0.52% -> +0.25% | Improved |
| use hybrid routing in lower layers instead of standard router (ablation) | HotMoE · Task conflict (ARC-C/RTE/BoolQ/CommonGen/WMT19) | MRD: -0.03% -> +0.25% | Improved |
| use token-expert routing in higher layers instead of standard router (ablation) | HotMoE · Task conflict (ARC-C/RTE/BoolQ/CommonGen/WMT19) | MRD: -0.26% -> +0.25% | Improved |
| add similarity-guided loss alongside GJS (ablation) | HotMoE · Task conflict (ARC-C/RTE/BoolQ/CommonGen/WMT19) | MRD: -0.44% -> +0.25% | Improved |
| add GJS loss alongside similarity-guided loss (ablation) | HotMoE · Task conflict (ARC-C/RTE/BoolQ/CommonGen/WMT19) | MRD: -0.85% -> +0.25% | Improved |
| add similarity-guided loss to MoLORA (ablation) | MoLORA · Task synergy (MRPC/RTE/HellaSwag/CommonGen/WMT19) | MRD: -0.33% -> +0.06% | Improved |
| add similarity-guided loss to standard Wg router (ablation) | HotMoE-Wg · Task synergy (MRPC/RTE/HellaSwag/CommonGen/WMT19) | MRD: -1.29% -> -0.79% | Improved |
| increase LoRA rank from 8 to 40 in multi-task LoRA | LoRA mix · Task synergy (MRPC/RTE/HellaSwag/CommonGen/WMT19) | AVG vs MRD: AVG 54.70 -> 56.13; MRD +0.24% -> -0.65% | Mixed |
| use HotMoE instead of HydraLoRA | multi-task MoE-LoRA · Task conflict (ARC-C/RTE/BoolQ/CommonGen/WMT19) | AVG score: 54.15 -> 54.05 | No clear effect |
| use HotMoE instead of best baseline on synergy tasks | multi-task MoE-LoRA · Task synergy (MRPC/RTE/HellaSwag/CommonGen/WMT19) | MRD: +0.24% -> +1.68% | Improved |
Closest papers by meaning
- FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts (NeurIPS 2025 · similarity 0.92)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.90)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.89)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
