Research map / Architectures and efficiency / Mixture of experts
Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
Qi Wang, Hanyang Peng, Yue Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add activation-based functional alignment (neuron permutation) (ablation) | Symphony-MoE 1.5Bx4 · ID average | average score: 33.94 -> 44.07 | Improved |
| add activation-based functional alignment (ablation) | Symphony-MoE 1.5Bx4 · MedCQA OOD | accuracy (%): 26.92 -> 35.26 | Improved |
| align all neurons instead of only 80% (ablation) | Symphony-MoE 1.5Bx4 · ID average | average score: 43.57 -> 44.07 | No clear effect |
| use SLERP instead of linear averaging for attention (ablation) | Symphony-MoE 1.5Bx4 · ID average | average score: 37.28 -> 44.07 | Improved |
| use selective linear (MergeKit) embedding merge instead of naive averaging (ablation) | Symphony-MoE 1.5Bx4 · ID average | average score: 40.19 -> 44.07 | Improved |
| sample calibration data from all expert domains instead of only general expert (ablation) | Symphony-MoE 1.5Bx4 · MedCQA OOD | accuracy (%): 29.44 -> 35.26 | Improved |
| Symphony-MoE vs Drop-Upcycling | MoE upcycling 0.5Bx4 · ID average | average score: 29.95 -> 31.77 | Improved |
| Symphony-MoE vs Drop-Upcycling | MoE upcycling 1.5Bx4 · MedCQA OOD | accuracy (%): 32.90 -> 35.26 | Improved |
| Symphony-MoE vs code expert on HumanEval | Symphony-MoE 1.5Bx4 · HumanEval | pass@1: 43.41 -> 42.39 | Hurt |
| use general-purpose M1 as anchor instead of other experts (ablation) | Symphony-MoE 1.5Bx4 · ID and OOD average | average score | Improved |
| increase number of experts from 1 to 4 (ablation) | Symphony-MoE 1.5Bx4 · ID and OOD | average score | Improved |
Closest papers by meaning
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.93)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.92)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.91)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
