Research map / Vision-language and multimodal / CLIP and zero-shot
Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies when and why visual prompt tuning (VPT) beats full finetuning across 19 VTAB-1k tasks, relating outcomes to task disparity, data distribution (FID), dataset size, overfitting, and optimization.
Problem
Unclear conditions under which visual prompt tuning outperforms full finetuning for transfer learning of large vision models, and the mechanisms behind its success.
Approach
Categorizes downstream tasks into four quadrants by task-objective disparity and FID-measured data distribution disparity, scales training set sizes, and tests overfitting and optimization hypotheses using Mixed and FT-then-PT variants, plus GradCAM/Integrated Gradients visualizations.
Key results
- VPT outperforms full finetuning on 16 of 19 VTAB-1k tasks with ViT-B/16 (ImageNet-21k), mean 79.53 vs 75.19 on Natural.
- With 400-10,000 training samples, FT surpasses VPT in 9 of 12 plotted cases as data grows; VPT wins 13 of 19 tasks in one-shot setting.
- Overfitting is observed mainly on Structured tasks; the Mixed and FT-then-PT variants do not beat FT, suggesting extra dimensions do not aid escaping local minima.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune visual prompts while freezing the backbone instead of full finetuning | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Natural | accuracy: 75.19 -> 79.53 | Helped |
| Tune visual prompts while freezing the backbone instead of full finetuning | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Specialized | accuracy: 83.48 -> 82.65 | Hurt |
| Tune visual prompts while freezing the backbone instead of full finetuning | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Structured | accuracy: 48.07 -> 56.09 | Helped |
| Add visual prompts and full-finetune the backbone (ablation) | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Natural | accuracy: 75.19 -> 72.80 | Hurt |
| Add visual prompts and full-finetune the backbone (ablation) | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Specialized | accuracy: 83.48 -> 79.78 | Hurt |
| Add visual prompts and full-finetune the backbone (ablation) | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Structured | accuracy: 48.07 -> 42.94 | Hurt |
| Full-finetune first, then tune added visual prompts (ablation) | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Natural | accuracy: 75.19 -> 78.20 | Helped |
| Full-finetune first, then tune added visual prompts (ablation) | ViT-B/16 pretrained on ImageNet-21k · VTAB-1k Structured | accuracy: 48.07 -> 50.38 | Helped |
| Use one-shot training rather than the standard VTAB-1k training split | VPT versus full finetuning · VTAB-1k Natural | accuracy: 41.66% vs 56.94% | Helped |
| Increase the training data available for downstream adaptation | Full finetuning versus VPT · VTAB-1k | accuracy | Mixed |
| Use Swin-Base instead of ViT-B/16 | VPT versus full finetuning · VTAB-1k Structured | accuracy: 59.65 -> 51.85 | Hurt |
| Use MAE pretraining instead of supervised ImageNet-21k pretraining | VPT versus full finetuning · VTAB-1k Structured | accuracy: 53.82 -> 27.91 | Hurt |
Surprising findings
- Adding extra dimensions via Mixed tuning (prompts plus full finetuning) performs worst, contrary to the local-minima escape hypothesis.
- Overfitting is not the main cause of FT's inferior performance: both methods show no overfitting on most Natural and Specialized tasks.
Limitations
- Experiments focus on image classification (VTAB-1k, FGVC); other tasks like segmentation are not explored.
- One-shot results are noisy and may not be conclusive, as the authors note.
Experimental setup
Datasets: VTAB-1k ImageNet-21k FGVC
Benchmarks: VTAB-1k FGVC
Metrics: accuracy number of wins FID
Baselines: full finetuning Mixed FT-then-PT linear probe
Builds on: visual prompt tuning VTAB-1k Fréchet Inception Distance
Closest papers by meaning
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.93)
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.92)
- Fair-VPT: Fair Visual Prompt Tuning for Image Classification (CVPR 2024 · similarity 0.91)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.90)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.