Research map / Vision-language and multimodal / CLIP and zero-shot
DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DA-VPT, which adds proxy-anchor metric learning losses that align visual prompts with same-class image tokens and the [CLS] token (plus optional bias tuning) to improve VPT-Deep on ViT recognition and segmentation.
Problem
Visual prompt tuning for ViTs learns prompts only from the task objective, leaving the distribution and class-relevant correlation between prompts and image/CLS tokens unexplored.
Approach
Treat deep-layer prompts as class proxies: map classes to prompts via k-means on class representations and apply a smoothed proxy-anchor metric loss between prompts and attention-output tokens (and between prompts and the CLS token), so prompts act as semantic bridges from image patches to the class token.
Key results
- On supervised ViT-B, DA-VPT+ reaches 91.94% mean FGVC accuracy versus 89.11% for VPT-Deep, and 76.14% mean VTAB-1K accuracy versus 71.96%.
- With MAE pre-training on ViT-B, DA-VPT+ gets 69.61% VTAB-1K mean accuracy versus 41.73% for VPT-Deep.
- On ADE20K with ViT-L, DA-VPT+ achieves 46.47 mIoU-SS versus 44.08 for VPT baseline, using about 13.7M tunable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DA-VPT semantic metric guidance and Efficient Bias | VPT-Deep on ViT-B with ImageNet-21K supervised pretraining · FGVC | mean accuracy: 2.83 percentage points | Helped |
| add DA-VPT semantic metric guidance and Efficient Bias | VPT-Deep on ViT-B with ImageNet-21K supervised pretraining · VTAB-1K | mean accuracy: 4.18 percentage points | Helped |
| add semantic metric losses and Efficient Bias | VPT-Deep · ADE20K and PASCAL Context | mIoU | Mixed |
| add both metric learning losses LML(x, p) and LML(p, xcls) (ablation) | DA-VPT · VTAB-1K Natural | accuracy: +1.08 pp | Helped |
| add both metric learning losses LML(x, p) and LML(p, xcls) (ablation) | DA-VPT · CUB-200-2011 | accuracy: +1.22 pp | Helped |
| add Efficient Bias (ablation) | DA-VPT with both metric losses · VTAB-1K Natural | accuracy: +0.97 pp | Helped |
| add Efficient Bias (ablation) | DA-VPT with both metric losses · CUB-200-2011 | accuracy: +1.03 pp | Helped |
| add LML(x, p) (ablation) | DA-VPT · VTAB-1K Natural | accuracy: +0.02 | No clear effect |
| add LML(p, xcls) (ablation) | DA-VPT · VTAB-1K Natural | accuracy: +0.06 | No clear effect |
| initialize prompts with dataset mean values (ablation) | DA-VPT · CUB-200-2011, Cars196, NABirds, and VTAB-Natural | top-1 accuracy | Hurt |
Surprising findings
- Mean-value prompt initialization hurts DA-VPT, since homogeneous prompts make it harder for metric guidance to learn discriminative information.
- Conventional metric learning losses (Proxy-NCA, triplet) led to training instability when applied to prompts, while Proxy-Anchor was stable.
Limitations
- Gains are reported to vary across pre-training methods, with layer-wise impact differing for MAE and MoCo.
- Combinations with other PEFT methods were explicitly left out of scope.
- Attention artifacts persist in prompt attention maps and their effect on performance is not resolved.
Experimental setup
Datasets: CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars VTAB-1k ADE20K PASCAL Context ImageNet-21K
Benchmarks: FGVC VTAB-1K ADE20K PASCAL Context
Metrics: top-1 accuracy mIoU-SS mIoU-MS parameter count
Baselines: VPT-Shallow VPT-Deep E2VPT GateVPT SPT-LoRA SPT-Adapter SSF SNF MoSA AdaptFormer Full fine-tuning Linear probing
Builds on: Visual Prompt Tuning (VPT) Proxy-Anchor loss Neighbourhood Components Analysis BitFit
Closest papers by meaning
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.92)
- Fair-VPT: Fair Visual Prompt Tuning for Image Classification (CVPR 2024 · similarity 0.91)
- Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning (ICLR 2024 · similarity 0.91)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.89)
- Visual Fourier Prompt Tuning (NeurIPS 2024 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.