Research map / Architectures and efficiency / Mixture of experts
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Merges fine-tuned ViT task models by keeping most weights merged via task arithmetic while upscaling each MLP into a weight-ensembling MoE whose input-conditioned router mixes task vectors, trained with test-time entropy minimization.
Problem
Merging multiple task-specific Transformer vision models into one multi-task model suffers from parameter interference and static merged solutions that cannot adapt to individual inputs.
Approach
Separate shared knowledge (pre-trained MLP weights) from task-specific knowledge (task vectors of MLPs) and dynamically combine them per input via a router that produces weights over task vectors, so the merged MLP is input-conditioned; the router is trained with unlabeled test-time entropy minimization.
Key results
- On eight-task merging of CLIP-ViT-B/32, WEMoE (2-layer router) reaches 89.4% average accuracy vs. 81.1% for AdaMerging (layer-wise) and 69.0% for Task Arithmetic.
- On CLIP-ViT-L/14, WEMoE achieves 93.6% average, slightly above traditional MTL (93.5%) and AdaMerging++ (91.0%).
- On corrupted test sets with CLIP-ViT-B/32, WEMoE (2-layer) gives strong gains on most distortions but drops sharply on pixelate (27.3% avg vs. 42.3% for AdaMerging).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace static merging with input-conditioned WEMoE | CLIP-ViT-B/32 model merging · Eight-task average | top-1 accuracy: 81.1 -> 89.4 | Helped |
| replace static merging with input-conditioned WEMoE | CLIP-ViT-L/14 model merging · Eight-task average | top-1 accuracy: 91.0 -> 93.6 | Helped |
| use a two-layer rather than zero-layer router (ablation) | WEMoE · Seen tasks (six-task average) | top-1 accuracy: 73.5 -> 86.5 | Helped |
| use a two-layer rather than zero-layer router (ablation) | WEMoE · Unseen tasks (MNIST, EuroSAT average) | top-1 accuracy: 64.2 -> 71.1 | Helped |
| use a two-layer rather than zero-layer router (ablation) | WEMoE · Clean test set (Cars, EuroSAT, RESISC45, GTSRB) | top-1 accuracy: 84.6 -> 93.2 | Helped |
| use a two-layer rather than zero-layer router (ablation) | WEMoE · Pixelate corrupted test set (Cars, EuroSAT, RESISC45, GTSRB) | top-1 accuracy: 40.7 -> 27.3 | Hurt |
| apply MoE up-scaling to attention and MLP weights separately instead of MLP only (ablation) | WEMoE · Eight tasks (CLIP-ViT-B/32) | top-1 accuracy: 88.3 -> 89.2 | Helped |
| apply MoE up-scaling to the entire Transformer block instead of MLP only (ablation) | WEMoE · Eight tasks (CLIP-ViT-B/32) | top-1 accuracy: 88.3 -> 86.4 | Hurt |
| use a two-layer rather than one-layer router (ablation) | WEMoE · Eight-task average (CLIP-ViT-B/32) | top-1 accuracy: 88.3 -> 89.4 | Helped |
| use a two-layer rather than zero-layer router (ablation) | WEMoE · Corrupted test distributions | top-1 accuracy | Mixed |
Surprising findings
- WEMoE with a 2-layer router slightly outperforms traditional multi-task learning on average despite merging methods having no access to training data.
- The 2-layer WEMoE overfits under severe pixelation corruption, performing worse than the 0-layer variant and AdaMerging on that distortion.
Limitations
- Upscaling MLPs into MoE increases total parameter count roughly linearly with number of tasks (e.g., 573.96M for 8 tasks on ViT-B/32).
- Performance degrades under severe image corruption such as pixelation, suggesting overfitting of the 2-layer router.
- Requires unlabeled test data for test-time entropy-minimization training of the router and is limited to classification tasks.
Experimental setup
Datasets: SUN397 Stanford Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD
Benchmarks: SUN397 Stanford Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD
Metrics: top-1 accuracy average accuracy
Baselines: Weight Averaging Fisher Merging RegMean Task Arithmetic Ties-Merging AdaMerging AdaMerging++ Traditional MTL
Builds on: task arithmetic AdaMerging Ties-Merging CLIP Mixture of Experts
Closest papers by meaning
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.88)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.88)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.88)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.87)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.87)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.