Research map / Label-efficient and robust learning / Recognition and long-tail

DePT: Decoupled Prompt Tuning

CVPR 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Multilingual language representation pretraining · 50 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DePT, a plug-and-play framework that adds a channel-adjusted transfer head to prompt tuning of CLIP to isolate base-specific knowledge, improving base-to-new and cross-dataset generalization across six prompt tuning baselines.

Problem

Prompt tuning of vision-language models suffers from the Base-New Tradeoff: gains on base (seen) classes degrade zero-shot generalization to new classes and datasets.

Approach

Attributes the tradeoff to a channel bias where base-specific knowledge occupies most feature channels; DePT adds a Channel Adjusted Transfer (CAT) head with channel-wise scaling/shift and a linear classifier to store base-specific knowledge in an isolated space, keeps the standard ITM head for zero-shot transfer, and fuses both heads at test time for base tasks.

Key results

• Averaged over 11 datasets, DePT improves CoOp from 75.18 to 77.29 harmonic mean (base 81.50->83.66, new 69.77->71.82).
• DePT improves all six baselines in base-to-new H, with absolute gains of 1.31% to 3.17% on base and 0.71% to 2.23% on new tasks averaged over 11 datasets.
• In cross-dataset generalization from ImageNet, DePT raises the source-dataset accuracy of all six baselines and improves average target accuracy in most cases.

Measured effects reported in the paper

Surprising findings

• Using only a CAT head with no ITM head (lambda=1.0) sharply degrades new-task accuracy, showing the ITM head is essential for zero-shot transfer.
• DePT gives little gain on OxfordPets, which the authors attribute to minimal category shift between CLIP pretraining and that dataset.

Limitations

• Hyperparameters (lambda, epochs, learning rate) are fixed across datasets and baselines, though optimal values differ, causing failure cases such as OxfordPets.
• Gains diminish when the downstream task is close to CLIP pretraining distribution.

Experimental setup

Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397 ImageNet-V2

Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397 ImageNet-V2

Metrics: base accuracy new accuracy harmonic mean accuracy top-1 accuracy

Baselines: CoOp CoCoOp KgCoOp MaPLe VPT PromptSRC CLIP-adapter

Builds on: CLIP CoOp CoCoOp MaPLe PromptSRC

Closest papers by meaning

• DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.86)
• DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection (ICML 2024 · similarity 0.82)
• TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model (CVPR 2024 · similarity 0.82)
• ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.82)
• EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.82)
• Leveraging First and Zeroth-Order Gradient to Address Imbalanced Black-Box Prompt Tuning via Minimax Optimization (AAAI 2025 · similarity 0.81)