Research map / Architectures and efficiency / Pruning and efficient tuning
TADFormer: Task-Adaptive Dynamic TransFormer for Efficient Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TADFormer, a PEFT framework for multi-task dense scene understanding that combines prompt-based task adaptation (TPC operator) with a Dynamic Task Filter conditioned on input context.
Problem
Parameter-efficient fine-tuning for multi-task learning often uses static task-specific parameters and fails to capture fine-grained, input-dependent task-specific features, limiting accuracy with few trainable parameters.
Approach
Task prompts produce task attention maps over patch tokens to create task-adapted features, which then pass through a Dynamic Task Filter that generates channel-wise convolution kernels from the input features via LoRA-style down-up projections, enabling input-conditioned task-specific adaptation with cross-task interaction.
Key results
- TADFormer (r=64) reaches +4.24% Δm over single-task full fine-tuning with 7.38M trainable parameters, versus MTLoRA (r=64) at +2.55% with 8.34M.
- TADFormer (r=16) achieves +2.44% Δm with 3.56M trainable parameters, versus MTLoRA (r=16) at +1.35% with 4.95M.
- Ablation: the DTF alone raises Δm by 1.65 points over the TADFormer baseline; removing the task attention map drops Δm by 0.39 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the low-rank setting from r=16 to r=64 | TADFormer · PASCAL-Context | relative improvement over single-task fine-tuning: +2.44 -> +4.24 | Helped |
| add task prompts and the TPC-operator without DTF (ablation) | TADFormer Baseline · PASCAL-Context | relative improvement over single-task fine-tuning: +1.3 -> +1.79 (+0.49) | Helped |
| add DTF without TPC-operator or task prompts (ablation) | TADFormer Baseline · PASCAL-Context | relative improvement over single-task fine-tuning: +1.3 -> +2.95 (+1.65) | Helped |
| add task prompts and DTF, without TPC-operator (ablation) | TADFormer Baseline · PASCAL-Context | relative improvement over single-task fine-tuning: +1.3 -> +3.24 (+1.94) | Helped |
| add TPC-operator and DTF together (ablation) | TADFormer Baseline · PASCAL-Context | relative improvement over single-task fine-tuning: +1.3 -> +3.63 (+2.33) | Helped |
| remove the TPC-operator based on the task attention map from the task-prompt-plus-DTF variant (ablation) | TADFormer · PASCAL-Context | relative improvement over single-task fine-tuning: 0.39 | Hurt |
| use TADFormer (r=64) instead of HyperFormer | Swin-T MTL model · PASCAL-Context test | mIoU (saliency): 65.54 -> 65.88 | Helped |
| use TADFormer (r=64) instead of HyperFormer | Swin-T MTL model · PASCAL-Context test | mIoU (semantic segmentation): 71.43 -> 70.82 | Hurt |
| use TADFormer (r=64) instead of HyperFormer | Swin-T MTL model · PASCAL-Context test | mIoU (human parts): 60.73 -> 60.45 | Hurt |
| use TADFormer (r=64) instead of MTLoRA (r=32) | Swin-T MTL model · PASCAL-Context test | rmse (surface normals): 16.59 -> 16.48 | Helped |
Surprising findings
- Adding only the DTF without task prompts or TPC operator already gives most of the gain (+2.95 Δm vs. +1.3 baseline), suggesting input-context filtering is the main contributor.
Limitations
- Evaluated mainly on PASCAL-Context with four dense prediction tasks, with a Swin-T backbone.
- Gains over MTLoRA are modest at low rank (r=16 vs 32 differences of around 1 point).
Experimental setup
Datasets: PASCAL-Context ImageNet-1k ImageNet-22k
Benchmarks: PASCAL-Context
Metrics: mIoU rmse relative multi-task improvement (Δm) trainable parameters (M)
Baselines: MTLoRA Polyhistor HyperFormer VL-Adapter LoRA Adapter Bitfit VPT Compacter Single-task full fine-tuning MTL full fine-tuning
Builds on: MTLoRA LoRA Visual Prompt Tuning Swin Transformer HRNet
Closest papers by meaning
- Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner (NeurIPS 2024 · similarity 0.90)
- DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations (CVPR 2025 · similarity 0.88)
- MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.87)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.87)
- Task-Adaptive Prompted Transformer for Cross-Domain Few-Shot Learning (AAAI 2024 · similarity 0.87)
- One Meta-tuned Transformer is What You Need for Few-shot Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.