Research map / Architectures and efficiency / Attention and transformers
Selective Visual Prompting in Vision Mamba
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Selective Visual Prompting (SVP), a token-wise input-dependent visual prompting method with dual Cross-Prompting and Inner-Prompting paths for parameter-efficient fine-tuning of Vision Mamba.
Problem
Visual prompting methods designed for ViT global attention do not suit Vision Mamba's sequential token-wise compression and propagation, so prefix prompts fail to activate its input and forget gates effectively for efficient downstream adaptation.
Approach
Generates input-dependent, token-level prompts via lightweight selective prompters added to each Vim token, which activates the selective SSM update and forget gates; a dual-path design shares prompts across layers (Cross-Prompting) and keeps layer-specific prompts (Inner-Prompting), balanced by learnable element-wise scales.
Key results
- On HTA with Vim-S, SVP reaches 89.8% average accuracy, versus 85.7% for DAM-VP and 83.2% for SPT on Vim-S.
- On VTAB-1K with Vim-S, SVP achieves 70.1% average accuracy versus 62.8% for VPT, and it beats full fine-tuning (67.1%).
- Ablation: Inner-Prompting alone and Cross-Prompting alone each improve over linear probing; combining both gives the best results on the four ablation datasets (e.g., CUB 82.8 vs 68.1 baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SVP instead of DAM-VP | Vim-S · HTA average | accuracy: +4.1% | Helped |
| Use SVP instead of VPT | Vim-S · HTA average | accuracy: +5.3% | Helped |
| Use SVP instead of VPT | Vim-S · VTAB-1K Natural | accuracy: +3.5% | Helped |
| Use SVP instead of VPT | Vim-S · VTAB-1K Specialized | accuracy: +4.7% | Helped |
| Use SVP instead of VPT | Vim-S · VTAB-1K Structured | accuracy: +13.8% | Helped |
| Replace pre-appended prompts with SVP's input-dependent token-wise prompts (ablation) | Vim-S · CIFAR, DTD, CUB, Flowers | accuracy: 5.2% | Helped |
| Place appended prompts in the middle rather than before image tokens (ablation) | Vim-S · CIFAR, DTD, CUB, Flowers | accuracy: 0.5% | Helped |
| Add Inner-Prompting to Cross-Prompting (ablation) | SVP on Vim-S · CIFAR, DTD, CUB, Flowers | accuracy | Helped |
| Add Cross-Prompting to Inner-Prompting (ablation) | SVP on Vim-S · CIFAR, DTD, CUB, Flowers | accuracy | Helped |
| Add Inner-Prompting to a frozen Vim-S with a learnable classifier (ablation) | Vim-S · CUB | accuracy: 14% | Helped |
| Add Cross-Prompting to a frozen Vim-S with a learnable classifier (ablation) | Vim-S · CIFAR, DTD, CUB, Flowers | accuracy | Helped |
| Increase the number of Cross-Prompting shared layers (ablation) | SVP on Vim-S · Classification benchmarks | accuracy | Mixed |
Surprising findings
- Frozen-backbone SVP outperforms full fine-tuning of Vim-S on HTA average (89.8 vs 88.5) and VTAB-1K average (70.1 vs 67.1).
Limitations
- Cross-Prompting sharing-layer count must be tuned per dataset (4, 8, or 12).
- Gains over ViT-B-based prompting methods are described as comparable rather than superior on HTA.
Experimental setup
Datasets: HTA VTAB-1K ImageNet-1K
Benchmarks: HTA VTAB-1K
Metrics: top-1 accuracy
Baselines: Full fine-tuning Linear probing VPT E2VPT DAM-VP SA2VP AutoVP SPT
Builds on: Vision Mamba VPT Mamba SPT
Closest papers by meaning
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.89)
- LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation (ICLR 2025 · similarity 0.89)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.89)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.88)
- Query Efficient Black-Box Visual Prompting with Subspace Learning (CVPR 2025 · similarity 0.88)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.