Research map / Architectures and efficiency / Pruning and efficient tuning
DTL: Disentangled Transfer Learning for Visual Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTL, a parameter-efficient transfer learning method that disentangles trainable low-rank side network from a frozen ViT/Swin backbone to cut GPU memory during fine-tuning while improving VTAB-1K accuracy.
Problem
Existing parameter-efficient transfer learning (PETL) methods reduce trainable parameters but not GPU memory during fine-tuning, because trainable modules are entangled with the backbone and require caching many intermediate activations for gradient propagation.
Approach
A lightweight Compact Side Network of low-rank linear mappings runs in parallel to the frozen backbone, extracting task-specific information and adding it back only to later blocks (from block M), so no gradients flow to early blocks; DTL+ adds a shared depthwise separable convolution and Swish activation with large beta.
Key results
- DTL+ reaches 77.7% average top-1 on VTAB-1K with ViT-B/16 using 0.05M trainable parameters and 1.7GB peak GPU memory, versus 75.7% for SSF and 75.6% for FacT (SSF at 4.9GB).
- DTL reduces GPU memory by about 65% on average versus full fine-tuning on ViT-B/16 and by 75% on Swin-B.
- On Swin-B, DTL+ outperforms FacT by about 1% average accuracy on VTAB-1K (78.4 vs 77.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a shared global DWConv after Swish in CSN (ablation) | DTL · VTAB-1K | Average accuracy: 76.7 -> 77.7 | Helped |
| Add Swish activation to the CSN output (ablation) | DTL with direct CSN output addition · VTAB-1K | Average accuracy: 76.0 -> 76.7 | Helped |
| Increase CSN rank from 2 to 4 (ablation) | DTL+ · VTAB-1K | Average accuracy: 77.7 -> 77.6 | Hurt |
| Decrease CSN rank from 2 to 1 (ablation) | DTL+ · VTAB-1K | Average accuracy: 77.7 -> 77.2 | Hurt |
| Adapt all backbone blocks rather than beginning adaptation at M=7 (ablation) | DTL+ · VTAB-1K | Average accuracy: 77.7 -> 77.9 | Mixed |
| Decrease the adaptation start index M from 12 to 11 (ablation) | DTL · VTAB-1K | Recognition accuracy: 75.9% -> 76.7% | Helped |
| Add DTL+ shared global DWConv | DTL · VTAB-1K | Average accuracy: +1.0 | Helped |
| Use DTL instead of the strongest PETL baseline | ViT-B/16 · VTAB-1K | Average accuracy: 75.7 -> 76.7 | Helped |
| Use DTL+ instead of NOAH | ViT-B/16 · Domain generalization: ImageNet-Sketch | Top-1 accuracy: 24.8 -> 35.7 | Helped |
| Use DTL+ instead of NOAH | ViT-B/16 · Domain generalization: ImageNet-A | Top-1 accuracy: 11.9 -> 14.2 | Helped |
| Use DTL instead of full fine-tuning | ViT-B/16 · VTAB-1K | Peak GPU memory: 4.7 -> 1.6 | Helped |
Surprising findings
- Adapting early backbone blocks (DTL+*) gives only 0.2% extra accuracy over DTL+ while raising GPU memory from 1.7GB to 3.1GB, suggesting limited cost-efficiency of early-feature adaptation.
Limitations
- Coarse granularity of interaction between backbone and trainable modules due to the disentangled design.
- Adapting early blocks increases GPU memory with limited accuracy gain, and larger-capacity variants may overfit under large domain gaps.
Experimental setup
Datasets: VTAB-1K ImageNet-1K Aircraft Pets Food-101 Cars Flowers102
Benchmarks: VTAB-1K ImageNet-Sketch ImageNet-V2 ImageNet-A ImageNet-R Aircraft Pets Food-101 Cars Flowers102
Metrics: top-1 accuracy peak GPU memory (GB) trainable parameters (M) throughput (imgs/sec)
Baselines: Full fine-tuning Linear probing BitFit VPT LST LoRA AdaptFormer NOAH FacT SSF
Builds on: LoRA AdaptFormer VPT LST NOAH
Closest papers by meaning
- V-PETL Bench: A Unified Visual Parameter-Efficient Transfer Learning Benchmark (NeurIPS 2024 · similarity 0.88)
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.84)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.84)
- Implicit Modeling for Transferability Estimation of Vision Foundation Models (NeurIPS 2025 · similarity 0.83)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.83)
- DiVE-k: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINED IMAGE RECOGNITION (ICLR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.