Research map / Architectures and efficiency / Mixture of experts
Mod-Squad: Designing Mixtures of Experts As Modular Multi-Task Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mutual-information training to encourage sparse task-expert assignments | Mod-Squad on Taskonomy · Taskonomy | expert-task assignment sparsity | Helped |
| combine MoE attention and MoE MLP rather than using only one (ablation) | Mod-Squad on Taskonomy · Taskonomy (13 tasks) | mean Δt (%): +3.9 / +3.8 -> +5.6 | Helped |
| prune task-unused experts (ablation) | Mod-Squad on PASCAL-Context · PASCAL-Context | mean Δt (%): +4.72 -> +4.65 | No clear effect |
| set attention Top-K to 2 instead of 6 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 3.5 -> -0.4 | Hurt |
| set attention Top-K to 9 instead of 6 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 3.5 -> 3.1 | Hurt |
| set MLP Top-K to 1 instead of 4 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 3.5 -> 3.3 | No clear effect |
| set MLP Top-K to 8 instead of 4 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 3.5 -> 3.1 | Hurt |
| increase attention experts from 6 to 15 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 2.0 -> 3.5 | Helped |
| increase attention experts from 15 to 18 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 3.5 -> 3.5 | No clear effect |
| increase MLP experts from 4 to 16 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 2.2 -> 3.5 | Helped |
| increase MLP experts from 16 to 20 (ablation) | Mod-Squad on Taskonomy · Taskonomy | mean Δt (%): 3.5 -> 3.5 | No clear effect |
| remove more experts from modified-MoE (ablation) | modified-MoE · Taskonomy | performance | Hurt |
Closest papers by meaning
- Towards Principled Task Grouping for Multi-Task Learning (NeurIPS 2026 · similarity 0.89)
- Merging Multi-Task Models via Weight-Ensembling Mixture of Experts (ICML 2024 · similarity 0.88)
- DMTG: One-Shot Differentiable Multi-Task Grouping (ICML 2024 · similarity 0.88)
- Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts (ICLR 2024 · similarity 0.87)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.87)
- Multi-Task Dense Prediction via Mixture of Low-Rank Experts (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.