Research map / Architectures and efficiency / Pruning and efficient tuning
VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VMT-Adapter (and a parameter-shared Lite variant) that shares down/up projections across tasks with lightweight task-specific modules, achieving near O(1) training/inference cost in task number for multi-task dense scene understanding on PASCAL-Context.
Problem
Parameter-efficient transfer learning for multi-task dense scene understanding: existing multi-task adapter methods use task-specific adapters, so parameters and training/inference cost grow linearly with the number of tasks.
Approach
Shared down- and up-projection adapters inserted in parallel to the MLP of each Swin layer capture cross-task knowledge, while small per-task scale-and-shift modules extract task-specific features; the Lite variant shares parameters across projections via sums of Kronecker products.
Key results
- VMT-Adapter achieves +3.96% average relative improvement over single-task full fine-tuning on PASCAL-Context with 1.13M encoder trainable parameters (3.68M total).
- VMT-Adapter-Lite achieves +1.34% improvement with 0.40M encoder parameters (2.95M total).
- Hyperformer, the strongest competing method, reaches +2.64% but uses 72.77M trainable parameters; Polyhistor reaches +2.34% with 6.41M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VMT-Adapter instead of single-task full fine-tuning | SwinTransformer-Tiny · PASCAL-Context four-task average | relative improvement: 0.00% -> 3.96% | Helped |
| use VMT-Adapter-Lite instead of single-task full fine-tuning | SwinTransformer-Tiny · PASCAL-Context four-task average | relative improvement: 0.00% -> 1.34% | Helped |
| share the adapter across tasks rather than use separate adapters | single-task adaptation methods · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: -2.71% -> -0.64% | Helped |
| set down-projection ratio rho=1 (ablation) | VMT-Adapter · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: 4.27% | Helped |
| set down-projection ratio rho=2 (ablation) | VMT-Adapter · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: 3.81% | Helped |
| set down-projection ratio rho=4 (ablation) | VMT-Adapter · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: 3.96% | Helped |
| set down-projection ratio rho=8 (ablation) | VMT-Adapter · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: 3.02% | Helped |
| increase parameter-sharing matrix dimension from m=3 to m=6 (ablation) | VMT-Adapter-Lite · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: 1.34% -> 0.61% | Hurt |
| increase parameter-sharing matrix dimension from m=6 to m=12 (ablation) | VMT-Adapter-Lite · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: 0.61% -> 0.06% | Hurt |
| use SwinTransformer-Base instead of SwinTransformer-Tiny (ablation) | VMT-Adapter · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: +3.96% -> +7.10% | Helped |
| use a High-Resolution decoder instead of an All-MLP decoder (ablation) | VMT-Adapter with SwinTransformer-Tiny · PASCAL-Context four-task average | relative improvement vs. Single-task Full Fine-tuning: +3.96% -> +2.41% | Hurt |
Surprising findings
- Shared Adapter outperforms Multiple Adapter, and reaches near parity with single-task full fine-tuning, suggesting cross-task collaboration helps.
Limitations
- Task-specific module parameters grow with task number and may exceed shared parameters when tasks reach thousands.
- Evaluated only on PASCAL-Context with four dense tasks.
Experimental setup
Datasets: PASCAL-Context
Benchmarks: PASCAL-Context
Metrics: mIoU mErr average relative improvement vs single-task full fine-tuning
Baselines: Single-task Full Fine-tuning Multi-task Full Fine-tuning Fine-tuning Decoders Multiple Adapter Shared Adapter Multiple LoRA Hyperformer Polyhistor Polyhistor-lite BitFit Relative bias
Builds on: Adapter LoRA Polyhistor Hyperformer Compactor
Closest papers by meaning
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.89)
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024 · similarity 0.88)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.88)
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.88)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.