Research map / Architectures and efficiency / Pruning and efficient tuning
Revisiting the Power of Prompt for Visual Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Prompt Tuning (SPT), which initializes visual prompt tokens from patch-embedding prototypes (K-means or cheap mean/max pooling/random sampling) of the downstream data, improving VPT especially under self-supervised pretraining.
Problem
Visual prompt tuning (VPT) is sensitive to prompt initialization and prompt length and performs poorly with self-supervised pre-trained ViTs.
Approach
Observes that prompt tokens gain normalized mutual information with patch tokens during VPT training, and therefore initializes prompts with downstream patch-token prototypes (K-means clusters, or cheaply via mean/max pooling or random sampling of a batch's tokens) so prompts start with high shared information.
Key results
- With MAE pre-training on ViT-B, SPT-Deep reaches 83.26% FGVC mean accuracy versus 72.02% for VPT-Deep and 82.80% for full fine-tuning, outperforming full fine-tuning in 19 of 24 cases with under 0.4% learnable parameters.
- With supervised ImageNet-21K pre-training, SPT-Deep achieves 91.40% FGVC mean accuracy, the best among compared PEFT methods, versus 89.11% for VPT-Deep.
- Random-sample and pooling initializations take about 43-65 seconds on CUB-200-2011, whereas K-means clustering takes about 27.3 days at 100% of the data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize prompts with downstream token prototypes rather than random prompts | SPT-Deep vs VPT-Deep · FGVC Mean Acc (5), MAE / ImageNet-1K | accuracy: 72.02 -> 83.26 | Helped |
| Initialize prompts with downstream token prototypes rather than random prompts | SPT-Deep vs VPT-Deep · FGVC Mean Acc (5), MoCo-v3 / ImageNet-1K | accuracy: 83.12 -> 86.00 | Helped |
| Use K-means token prototypes instead of random initialization (ablation) | SPT-Deep · CUB-200-2011, supervised ImageNet-21K, prompts trainable | accuracy: 88.5 -> 91.2 (+2.7) | Helped |
| Initialize prompts by randomly sampling inferred patch tokens instead of random initialization (ablation) | SPT-Deep · CUB-200-2011, MAE / ImageNet-1K, prompts trainable | accuracy: 68.3 -> 80.1 (+11.8) | Helped |
| Initialize prompts by mean pooling inferred patch tokens and freeze prompts during adaptation (ablation) | SPT-Deep · CUB-200-2011, MAE / ImageNet-1K, frozen prompts | accuracy: 25.3 -> 37.7 (+12.4) | Helped |
| Use same-level features for each layer's prompt initialization instead of first-layer features at every layer (ablation) | SPT-Deep · CUB-200-2011, MAE / ImageNet-1K | accuracy: 76.5 -> 80.1 | Helped |
| Increase the prompt length (ablation) | SPT-Deep · CUB-200-2011, MAE / ImageNet-1K | accuracy | Helped |
| Construct prompts from other tasks' data rather than the downstream task's data (ablation) | SPT · FGVC benchmarks | accuracy | Hurt |
| Scale the backbone from ViT-B toward ViT-H (ablation) | SPT vs VPT · FGVC benchmarks | accuracy: up to 15.5% | Helped |
| Use K-means rather than mean pooling for prompt construction (ablation) | SPT-Deep · CUB-200-2011 prompt construction time, 100% training data | time: 42.63s -> 27.30d | Hurt |
Surprising findings
- Freezing the prompts after self-prompt initialization and training only the task head still yields large gains over random-initialized VPT (e.g., +13.8 on CUB-200 with MAE).
Limitations
- No rigorous theoretical guarantee for the proposed initialization strategy (stated by authors).
- MAE-pretrained backbones remain less adaptable than MoCo-v3 according to the authors.
- Experiments are limited to image classification benchmarks.
Experimental setup
Datasets: CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars VTAB-1K ImageNet-1K iNaturalist 2018 ImageNet-21K
Benchmarks: CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars VTAB-1K
Metrics: top-1 accuracy mean accuracy trainable parameters (M)
Baselines: Full fine-tuning VPT-Shallow VPT-Deep GateVPT Linear probing Bias tuning Adapter MP SSF SNF E2VPT
Builds on: Visual Prompt Tuning (VPT) Masked Autoencoder (MAE) Vision Transformer (ViT) GateVPT E2VPT
Closest papers by meaning
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.93)
- VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.92)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.90)
- Query Efficient Black-Box Visual Prompting with Subspace Learning (CVPR 2025 · similarity 0.89)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.