Research map / Architectures and efficiency / Pruning and efficient tuning
DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiTASK, which freezes the SVD singular vectors of pre-trained ViT/Swin weights and adapts only singular values via CPAB neural diffeomorphisms, enabling parameter-efficient multi-task dense prediction fine-tuning.
Problem
Parameter-efficient fine-tuning of pre-trained vision transformers for multi-task learning suffers from task interference when low-rank updates force tasks into shared constrained subspaces, degrading performance relative to single-task tuning.
Approach
Keep the pre-trained left/right singular vectors of each weight matrix fixed and learn a CPAB diffeomorphism that transforms the singular values, with a shared joint module plus task-specific modules, requiring only about 32 parameters per layer.
Key results
- DiTASK achieves +3.22% relative improvement over single-task tuning on PASCAL-Context with 1.61M trainable encoder parameters, versus MTLoRA (r=64) at +2.55% with 6.40M.
- DiTASK reports a 26.27% improvement in delta-m over MTLoRA while using 4x fewer parameters, and on NYUD achieves +20.65% delta-m vs MTLoRA +8.42%.
- Batch runtime drops from 35.20s (MTLoRA) to 27.31s (DiTASK), with 8.5K adaptation parameters versus 4.86M for MTLoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply joint and task-specific diffeomorphic singular-value modulation while preserving singular vectors | Swin-Tiny with DITASK · PASCAL-MTL | relative improvement over single-task tuning: +3.22 | Helped |
| apply DITASK instead of MTLoRA | Swin-Tiny multi-task adaptation · PASCAL-MTL | relative improvement over single-task tuning: 26.27% | Helped |
| apply DITASK instead of MTLoRA | Swin-Tiny multi-task adaptation · NYUD | MIOU: 37.18 -> 43.85 | Helped |
| apply DITASK instead of MTLoRA | Swin-Tiny multi-task adaptation · NYUD | RMSE: 0.635 -> 0.606 | Helped |
| use joint singular-value modulation alone (ablation) | DITASK · PASCAL-MTL | relative improvement over single-task tuning: 2.77% | Helped |
| use task-wise singular-value modulation alone (ablation) | DITASK · PASCAL-MTL | relative improvement over single-task tuning: 2.85% | Helped |
| combine joint and task-wise singular-value modulation rather than using either alone (ablation) | DITASK · PASCAL-MTL | relative improvement over single-task tuning: 3.22% | Helped |
| scale the backbone from Swin-Tiny to larger Swin variants | DITASK · PASCAL-MTL | relative improvement over single-task tuning: +3.22 -> +4.68 -> +6.52 -> +7.79 | Helped |
Surprising findings
- Preserving singular vectors and only modulating singular values with a diffeomorphism outperforms full multi-task fine-tuning (+3.22% vs +2.23%) and decoder-only tuning, which is far worse (-9.95%).
- Single-task DiTASK (+5.33%) outperforms the MTL variant (+3.22%) on PASCAL-Context.
Limitations
- Results reported on a single backbone family (Swin) and two dense prediction datasets.
- Paper's ablation of joint vs task-wise modulation shows modest gains (2.77% to 3.22%).
- Differences in delta-m versus several baselines are reported from single runs with no variance stated in main tables.
Experimental setup
Datasets: PASCAL-Context (PASCAL MTL) NYUD ImageNet-21k (pre-training)
Benchmarks: PASCAL-Context NYUD
Metrics: mIoU RMSE delta-m (relative improvement over single-task) trainable parameters batch runtime max GPU memory
Baselines: MTLoRA LoRA Adapter VL-Adapter Polyhistor HyperFormer Compactor BitFit VPT DoRA ReFT SVFT
Builds on: MTLoRA LoRA CPAB transformations MTINet
Closest papers by meaning
- TADFormer: Task-Adaptive Dynamic TransFormer for Efficient Multi-Task Learning (CVPR 2025 · similarity 0.88)
- Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic (ICLR 2025 · similarity 0.86)
- Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks (CVPR 2024 · similarity 0.86)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.86)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.85)
- Task-Conditioned Adaptation of Visual Features in Multi-Task Policy Learning (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.