Research map / Architectures and efficiency / Mixture of experts
Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMTAL, which decomposes a pre-trained ViT's FFN into low-rank MoE experts with LoRA, adds a Quality Retaining distillation loss for asynchronous multi-task learning, and fades the router so the learned weights reparameterize back into the original ViT.
Problem
Multi-task learning with Vision Transformers suffers from inefficient MoE/LoRA combinations that hamper expert optimization and LoRA reparameterization, plus synchronized multi-task optimization that disrupts tasks' natural convergence paces.
Approach
Clusters FFN up-projection columns (balanced k-means) into low-rank experts with LoRA adapters, uses a knowledge bank of EMA class logits with loss-reciprocal-weighted KL distillation to retain converged tasks' knowledge, and gradually fades the router so the experts merge back into the dense ViT with no inference overhead.
Key results
- EMTAL-4 reaches 91.73% mean top-1 accuracy on Multi-task FGVC with 1.20M tunable parameters and no extra inference cost, versus 87.49% for separate full fine-tuning.
- On Specialized VTAB-1k, EMTAL-4 achieves 87.89% mean accuracy, above MOELoRA's 87.01%.
- On NYUv2 with TaskPrompter, adding EMTAL raises semseg mIoU from 50.40 to 52.90 and gives a mean relative gain of 1.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMTAL-4 instead of Aligned-MTL | ViT-B/16 multi-task learner · Multi-task FGVC | Mean top-1 accuracy: 90.17 -> 91.73 | Helped |
| Add EMTAL to TaskPrompter-Base | TaskPrompter-Base · NYUv2 | Semseg mIoU: 50.40 -> 52.90 | Helped |
| Add EMTAL to TaskPrompter-Base | TaskPrompter-Base · NYUv2 | Depth RMSE: 0.5402 -> 0.5284 | Helped |
| Add EMTAL to TaskPrompter-Base | TaskPrompter-Base · NYUv2 | Normal mErr: 18.91 -> 18.95 | Hurt |
| Add EMTAL to TaskPrompter-Base | TaskPrompter-Base · NYUv2 | Boundary odsF: 77.60 -> 77.10 | Hurt |
| Add MoEfied LoRA (ablation) | Union FT · Multi-task FGVC | Mean top-1 accuracy: 87.39 -> 90.27 | Helped |
| Add Quality Retaining (ablation) | MoEfied LoRA · Multi-task FGVC | Mean top-1 accuracy: 90.27 -> 91.73 | Helped |
| Add MoEfied LoRA (ablation) | QR without MoEfied LoRA · Multi-task FGVC | Mean top-1 accuracy: 91.04 -> 91.73 | Helped |
| Add router fading (ablation) | MoEfied LoRA with LoRA · Multi-task FGVC | Inference time: 13.87 -> 7.15 | Helped |
| Set the expert cluster count to 16 instead of 1 (ablation) | MoEfied LoRA · Multi-task FGVC | Mean top-1 accuracy: 88.83 -> 90.27 | Helped |
| Set the expert cluster count to 64 instead of 16 (ablation) | MoEfied LoRA · Multi-task FGVC | Mean top-1 accuracy: 90.27 -> 89.34 | Hurt |
| Construct experts using the proposed weight-similarity clustering instead of co-activation clustering (ablation) | MoEfied LoRA · Multi-task FGVC | Mean top-1 accuracy: 89.30 -> 90.27 | Helped |
Surprising findings
- Ablation on cluster count shows 16 clusters is optimal; both 1 and 192 clusters reduce accuracy.
- The paper reports that the optimal LoRA rank differs by task (rank 1 best for CUB, rank 4 for Stanford Cars), so a unified rank is suboptimal.
Limitations
- A unified LoRA rank across all experts is used, though tasks and layers need different optimal ranks.
- Evaluation is limited to ViT-B/16 image classification benchmarks and one dense prediction setup.
- Code is promised only upon acceptance, not released.
Experimental setup
Datasets: CUB-200-2011 Stanford Cars FGVC-Aircraft Oxford Flowers Specialized VTAB-1k NYUv2 ImageNet-21K
Benchmarks: Multi-task FGVC Specialized VTAB-1k NYUv2
Metrics: top-1 accuracy mean accuracy tunable parameters (M) inference time (ms) mIoU RMSE mean Delta
Baselines: Separate FT Union FT Nash-MTL Aligned-MTL GLS AMTL Dual-Prompt Erudite MLoRE MOELoRA
Builds on: MoEfication LoRA Erudite MOELoRA AMTL
Closest papers by meaning
- TADFormer: Task-Adaptive Dynamic TransFormer for Efficient Multi-Task Learning (CVPR 2025 · similarity 0.90)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.87)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.86)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.86)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.