Research map / Architectures and efficiency / Mixture of experts
Multi-Task Dense Prediction via Mixture of Low-Rank Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLoRE, a decoder-side mixture-of-experts for multi-task dense prediction that adds a task-shared generic convolution path and linear low-rank experts that re-parameterize into a single conv at inference.
Problem
Mixture-of-experts decoders for multi-task dense prediction struggle to model global relations among all tasks and adding experts greatly increases parameters and FLOPs.
Approach
Add a task-sharing generic 3x3 convolution path in parallel with MoE so all tasks share explicit parameters, and make experts low-rank (3x3 then 1x1 conv, no activations) so many experts are cheap and can be re-parameterized into one conv per task at inference.
Key results
- On PASCAL-Context with ViT-large, MLoRE reaches 81.41 semseg mIoU, 70.52 parsing mIoU, and 75.42 boundary odsF, exceeding TaskExpert (80.64, 69.42, 73.30).
- On NYUD-v2, MLoRE achieves 55.96 semseg mIoU, 0.5076 depth RMSE, 18.33 normals mErr, and 78.43 boundary odsF, the best on all four tasks.
- Low-rank experts cut parameters by over 60% versus standard MoE (e.g., 15 experts: 6.3M vs 29.9M for 3x3 setting per Table 1 comparisons) and the efficient ViT-small variant uses about 35% of TaskExpert's GFLOPs (72G vs 204G).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the task-sharing generic convolution path to the low-rank expert path (ablation) | MLoRE on PASCAL-Context · PASCAL-Context | MTL ∆m: -1.71 -> -0.88 | Helped |
| add a task-specific low-rank expert path (ablation) | MLoRE on PASCAL-Context · PASCAL-Context | MTL ∆m: -0.88 -> -0.58 | Helped |
| replace standard MoE experts with low-rank expert networks (ablation) | ViT-base baseline on PASCAL-Context · PASCAL-Context | MTL ∆m: -1.20 -> -1.71 | Hurt |
| use varied task-sharing expert ranks from 16 to 128 instead of uniform rank 16 (ablation) | MLoRE on PASCAL-Context · PASCAL-Context | MTL ∆m: -0.77 -> -0.58 | Helped |
| use varied task-sharing expert ranks from 16 to 128 instead of uniform rank 128 (ablation) | MLoRE on PASCAL-Context · PASCAL-Context | MTL ∆m: -1.07 -> -0.58 | Helped |
| add a position-aware branch to the basic router (ablation) | MLoRE on PASCAL-Context · PASCAL-Context | MTL ∆m: -0.75 -> -0.58 | Helped |
| replace the basic router with a position-only router (ablation) | MLoRE on PASCAL-Context · PASCAL-Context | MTL ∆m: -0.75 -> -0.82 | Hurt |
| replace sample-dependent router input with learnable parameters (ablation) | MLoRE on PASCAL-Context · PASCAL-Context | MTL ∆m: -0.75 -> -0.91 | Hurt |
| activate 60% of experts rather than all expert networks (ablation) | MLoRE · PASCAL-Context | MTL gain | Helped |
| increase the number of task-sharing experts to 15 (ablation) | MLoRE · PASCAL-Context | MTL gain | Helped |
| use the full MLoRE method instead of TaskExpert | ViT-large multi-task model · PASCAL-Context | mIoU: 80.89 -> 81.41 | Helped |
| use the full MLoRE method instead of TaskExpert | ViT-large multi-task model · PASCAL-Context | odsF: 73.30 -> 75.42 | Helped |
Surprising findings
- Without the generic path, almost no expert is activated by all tasks in one sample, supporting the claim that standard decoder MoE fails to capture global task relations.
- Activating all experts hurts performance noticeably, so sparsity (about 60% of experts per task) matters for feature discrimination.
Limitations
- Evaluated only on two indoor/scene multi-task datasets (PASCAL-Context and NYUD-v2).
- Stop-gradient on the task-sharing generic path is a heuristic choice motivated by empirical gradient conflict observations.
- Compute hardware is not stated in the text.
Experimental setup
Datasets: PASCAL-Context NYUD-v2
Benchmarks: PASCAL-Context NYUD-v2
Metrics: mIoU maxF mErr odsF RMSE MTL gain delta-m FLOPs parameters
Baselines: PAD-Net MTI-Net ATRC InvPT TaskPrompter TaskExpert M3ViT Mod-Squad
Builds on: Mixture of Experts LoRA TaskPrompter InvPT M3ViT
Closest papers by meaning
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.89)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.87)
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.87)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.87)
- Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion (NeurIPS 2024 · similarity 0.87)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.