Research map / Architectures and efficiency / Pruning and efficient tuning

MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning

CVPR 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 33 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MTLoRA, a parameter-efficient fine-tuning framework for multi-task dense prediction that uses shared task-agnostic and task-specific low-rank adapters on a Swin backbone, plus a lighter MTLoRA+ variant.

Problem

Parameter-efficient fine-tuning has been studied for single-task adaptation, but adapting a shared backbone in multi-task learning models with few trainable parameters while avoiding task conflicts and keeping single-pass inference is largely unexplored.

Approach

Combine task-agnostic LoRA modules shared across all tasks in most transformer blocks with task-specific LoRA modules in the last block of each stage, so the backbone runs once for all tasks while task conflicts are disentangled; also add LoRA to patch-merging layers (MTLoRA+).

Key results

• MTLoRA with Swin-Tiny achieves +2.55% Δm over single-task fine-tuning with 8.34M trainable parameters, versus +2.23% for full MTL fine-tuning with 30.06M parameters (3.6x reduction).
• MTLoRA+ (r=16) reaches +1.19% Δm with 4.29M trainable parameters, while Polyhistor reaches +2.34% with 8.96M but requires separate per-task execution.
• Removing the task-specific modules lowers the accuracy-efficiency trade-off; QKV low-rank adaptation is the most important location per the ablation (Δm drops to +1.26% without it).

Measured effects reported in the paper

Surprising findings

• Parameter-efficient MTLoRA with 3.6x fewer trainable parameters outperforms full fine-tuning of the MTL model on Δm.
• Adding low-rank modules to patch-merging layers (MTLoRA+) is a practice not previously explored and improves the accuracy-efficiency trade-off.

Limitations

• Evaluated only on PASCAL-Context with four dense tasks.
• Gains over single-task baselines are reported as averages with modest margins; MTLoRA with r=16 does not beat full fine-tuning on Δm (+1.35% vs +2.23%).

Experimental setup

Datasets: PASCAL-Context ImageNet-1K ImageNet-22K

Benchmarks: PASCAL-Context

Metrics: mIoU rmse delta m (%) trainable parameters (M)

Baselines: Single Task fine-tuning MTL full fine-tuning Adapter Bitfit VPT Compacter LoRA VL-Adapter HyperFormer Polyhistor

Builds on: LoRA Swin Transformer PolyHistor HyperFormer

Closest papers by meaning

• Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters (CVPR 2025 · similarity 0.88)
• Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.88)
• SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.88)
• RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.88)
• TADFormer: Task-Adaptive Dynamic TransFormer for Efficient Multi-Task Learning (CVPR 2025 · similarity 0.87)
• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective (NeurIPS 2025 · similarity 0.87)