Research map / Architectures and efficiency / Mixture of experts
M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
hanxue liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dense ViT feed-forward layers with task-dependent sparse MoE | M-ViT · PASCAL-Context | Δm (%): -1.77 -> +2.71 | Improved |
| Replace dense ViT feed-forward layers with task-dependent sparse MoE | M-ViT · NYUD-v2 | Δm (%): −6.27 -> +1.59 | Improved |
| Add FPGA computation reordering to MoE inference | MoE ViT-small on FPGA · PASCAL-Context | Energy (W·s): 6.375 -> 0.690 | Improved |
| Add FPGA computation reordering to MoE inference | MoE ViT-small on FPGA · NYUD-v2 | Energy (W·s): 7.506 -> 0.845 | Improved |
| Replace a single shared router with separate task-specific routers (ablation) | M³ViT · PASCAL-Context | Δm (%): +1.40 -> +2.71 | Improved |
| Replace a single shared router with separate task-specific routers (ablation) | M³ViT · NYUD-v2 | Δm (%): +0.31 -> +1.59 | Improved |
| Use task-conditioned shared routing instead of separate task routers (ablation) | M³ViT · PASCAL-Context | Δm (%): +2.71 -> +2.22 | Hurt |
| Use task-conditioned shared routing instead of separate task routers (ablation) | M³ViT · NYUD-v2 | Δm (%): +1.59 -> +0.74 | Hurt |
| Remove computation reordering and use cache-based expert loading (ablation) | MoE ViT-small on FPGA · PASCAL-Context | Energy (W·s): 0.690 -> 6.375 | Hurt |
| Remove computation reordering and use cache-based expert loading (ablation) | MoE ViT-small on FPGA · NYUD-v2 | Energy (W·s): 0.845 -> 7.506 | Hurt |
| Scale the MoE ViT backbone from ViT-small to ViT-base | M³ViT · PASCAL-Context | Δm (%): +2.71 -> +4.00 | Improved |
| Scale the MoE ViT backbone from ViT-small to ViT-base | M³ViT · NYUD-v2 | Δm (%): +1.59 -> +8.32 | Improved |
Later papers that compared with this method
2 later accepted papers report M³ViT as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- Mod-Squad: Designing Mixtures of Experts As Modular Multi-Task Learners (CVPR 2023 · better on 6 of 6 shared results)
- Token Turing Machines (CVPR 2023 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner (NeurIPS 2024 · similarity 0.90)
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · similarity 0.88)
- TADFormer: Task-Adaptive Dynamic TransFormer for Efficient Multi-Task Learning (CVPR 2025 · similarity 0.87)
- Merging Multi-Task Models via Weight-Ensembling Mixture of Experts (ICML 2024 · similarity 0.87)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.87)
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
