Research map / Architectures and efficiency / Mixture of experts
Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
Li Shen, Anke Tang, Enneng Yang, Guibing Guo, Yong Luo, Lefei Zhang, Xiaochun Cao, Bo Du, Dacheng Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WEMoE dynamic merging instead of static AdaMerging (layer) | CLIP-ViT-B/32 · 8 tasks avg | average accuracy (%): 80.9 -> 89.4 | Improved |
| use WEMoE instead of traditional MTL | CLIP-ViT-B/32 · 8 tasks avg | average accuracy (%): 88.9 -> 89.4 | Improved |
| prune MLP task vectors to 90% sparsity and share routing across layers (E-WEMoE-90%) | WEMoE · 8 tasks avg (ViT-L/14) | average accuracy (%): 93.6 -> 93.5 | No clear effect |
| increase task vector sparsity from 90% to 99% (ablation) | E-WEMoE · 8 tasks avg (ViT-B/32) | average accuracy (%): 89.1 -> 86.8 | Hurt |
| add 2 routing layers instead of 0 (ablation) | WEMoE · 6 seen tasks avg (ViT-B/32) | average accuracy (%): 73.5 -> 86.5 | Improved |
| add 2 routing layers instead of 0 (ablation) | WEMoE · 2 unseen tasks avg (ViT-B/32) | average accuracy (%): 64.2 -> 71.1 | Improved |
| use WEMoE instead of AdaMerging on unseen tasks | CLIP-ViT-B/32 · 2 unseen tasks avg | average accuracy (%): 70.0 -> 71.1 | Improved |
| use E-WEMoE instead of WEMoE on unseen tasks | WEMoE · 2 unseen tasks avg (ViT-B/32) | average accuracy (%): 71.1 -> 61.7 | Hurt |
| use WEMoE instead of AdaMerging under motion blur | CLIP-ViT-B/32 · Motion blur corrupted test | average accuracy (%): 80.6 -> 87.6 | Improved |
| use WEMoE under pixelate corruption | CLIP-ViT-B/32 · Pixelate corrupted test | average accuracy (%): 42.3 -> 27.3 | Hurt |
| upcycle entire transformer block instead of MLP only (ablation) | WEMoE · Up-scaling strategy, 8 tasks | average accuracy (%): 88.3 -> 86.4 | Hurt |
| upcycle attention and MLP separately instead of MLP only (ablation) | WEMoE · Up-scaling strategy, 8 tasks | average accuracy (%): 88.3 -> 89.2 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Ties-Merging — TIES-Merging: Resolving Interference When Merging Models (NeurIPS 2023) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Merging Multi-Task Models via Weight-Ensembling Mixture of Experts (ICML 2024 · similarity 0.92)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.90)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.90)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.90)
- CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging (ICML 2025 · similarity 0.89)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
