Research map / Vision-language and multimodal / CLIP and zero-shot
SA²VP: Spatially Aligned-and-Adapted Visual Prompt
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SA2VP, a visual prompt tuning method that learns a 2D prompt token map spatially aligned with the image token map, using siamese dual-pathway spatially aligned cross attention and a prompt adapter, for parameter-efficient adaptation of ViT.
Problem
Sequential visual prompt tuning (e.g., VPT) learns unordered prompt tokens that cannot model spatial relations in images and cannot prompt different image tokens individually, limiting parameter-efficient adaptation of large vision models.
Approach
Replace the unordered prompt sequence with a 2D prompt token map of the same (or scaled) size as the image token map, with pre-learned positional embeddings added, so each prompt token spatially aligns with and prompts only its corresponding image tokens via a local c×c window spatially aligned cross attention between a prompt pathway and a base pathway.
Key results
- On FGVC with ViT-B/16, SA2VP reaches 90.08% mean accuracy versus 89.22% for E2VPT and 89.11% for VPT-deep, using 0.99% tuned parameters.
- On HTA (10 datasets), SA2VP achieves 91.5% mean accuracy, above DAM-VP at 88.5%.
- On VTAB-1k, SA2VP achieves 80.97 Natural, 85.73 Specialized, and 60.80 Structured mean accuracy, the best of compared methods on all three groups.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SA2VP instead of E2VPT | ViT-B/16 visual prompting · FGVC (mean) | accuracy: 89.22 -> 90.08 | Helped |
| use SA2VP instead of AdaptFormer | ViT-B/16 fine-tuning · HTA (mean) | accuracy: 89.0 -> 91.5 | Helped |
| use SA2VP instead of AdaptFormer | ViT-B/16 fine-tuning · VTAB-1k Natural | accuracy: 80.56 -> 80.97 | Helped |
| use SA2VP instead of AdaptFormer | ViT-B/16 fine-tuning · VTAB-1k Specialized | accuracy: 84.88 -> 85.73 | Helped |
| use SA2VP instead of LoRA | ViT-B/16 fine-tuning · VTAB-1k Structured | accuracy: 59.78 -> 60.80 | Helped |
| use SA2VP instead of E2VPT | Swin Transformer visual prompting · VTAB-1k Natural | accuracy: 83.31 -> 80.81 | Hurt |
| use SA2VP instead of Full Fine-tune | Swin Transformer fine-tuning · VTAB-1k Specialized | accuracy: 86.21 -> 86.30 | Helped |
| use SA2VP instead of Full Fine-tune | Swin Transformer fine-tuning · VTAB-1k Structured | accuracy: 59.65 -> 60.03 | Helped |
| add spatially aligned cross attention instead of global cross attention (ablation) | SA2VP · VTAB-1k Natural | accuracy: 79.37 -> 80.97 | Helped |
| add spatially aligned cross attention instead of global cross attention (ablation) | SA2VP · VTAB-1k Specialized | accuracy: 83.65 -> 85.73 | Helped |
| add spatially aligned cross attention instead of global cross attention (ablation) | SA2VP · VTAB-1k Structured | accuracy: 56.98 -> 60.80 | Helped |
| add the prompt adapter (ablation) | SA2VP · VTAB-1k Specialized | accuracy: 82.63 -> 85.73 | Helped |
Limitations
- Ablations and comparisons are reported mainly on VTAB-1k; the Swin Transformer evaluation covers only VTAB-1k groups.
- Hyperparameters such as window size c, balancing weights γ and λ are tuned empirically, with no sensitivity analysis reported in the main text.
Experimental setup
Datasets: ImageNet-21K CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars CIFAR-10 CIFAR-100 DTD Food-101 GTSRB SVHN
Benchmarks: FGVC HTA VTAB-1k
Metrics: classification accuracy
Baselines: Full Fine-tune Head Fine-tune AdaptFormer LoRA VPT-shallow VPT-deep E2VPT EXPRES DAM-VP
Builds on: VPT AdaptFormer SSF E2VPT DAM-VP
Closest papers by meaning
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.92)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.91)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.90)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.90)
- VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.