Research map / Label-efficient and robust learning / Self-supervised and contrastive
VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIPAMIN, a parameter-free visual prompt initialization for self-supervised ViTs that combines attention-guided token matching with orthogonal injection outside the frozen self-attention subspace, improving VPT on VTAB-1k and few-shot FGVC.
Problem
Visual prompt tuning (VPT) with self-supervised backbones yields near-uniform prompt attention and prompt outputs collapsing into the frozen self-attention subspace, hurting adaptation under distribution shift and in few-shot settings.
Approach
Initialize each prompt by averaging the top-k input tokens most similar to a random prompt in the key space (matching module), then mix in a component projected orthogonal to the row space of the frozen self-attention output via pseudoinverse of W_V (orthogonalizing module); requires only one forward pass and no extra parameters.
Key results
- On MoCo-v3 ViT-B/16 VTAB-1k, VIPAMIN reaches 69.86 mean accuracy vs. 68.33 for SPT and 57.94 for VPT, and 66.23 for full fine-tuning.
- On MAE ViT-B/16 VTAB-1k, VIPAMIN reaches 64.09 mean accuracy vs. 62.58 for SPT and 61.28 for full fine-tuning.
- In 1-shot to 8-shot FGVC, VIPAMIN outperforms SPT/rand in mean accuracy by roughly 1-2 points, and VPT by 7.7 points at k=1 and 24.7 points at k=8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIPAMIN instead of SPT | MoCo-v3 ViT-B/16 visual prompt tuning · VTAB-1k Mean | accuracy: 68.33 -> 69.86 | Helped |
| Use VIPAMIN instead of SPT | MAE ViT-B/16 visual prompt tuning · VTAB-1k Mean | accuracy: 62.58 -> 64.09 | Helped |
| Use VIPAMIN instead of SPT/rand | MoCo-v3 ViT-B/16 visual prompt tuning · FGVC few-shot benchmarks | mean accuracy: 23.7 -> 25.8 (1-shot); 36.5 -> 38.2 (2-shot); 51.7 -> 52.4 (4-shot); 65.5 -> 66.4 (8-shot) | Helped |
| Use VIPAMIN instead of SPT-Deep | MoCo-v3 ViT-B/16 VPT-Deep · Oxford Flowers102 | accuracy: 93.2 -> 94.0 | Helped |
| Add the matching module to SPT initialization (ablation) | MoCo-v3 ViT-B/16 visual prompt tuning · VTAB-1k Natural | accuracy: 74.47 -> 76.50 | Helped |
| Add the matching module to SPT initialization (ablation) | MoCo-v3 ViT-B/16 visual prompt tuning · VTAB-1k Specialized | accuracy: 83.93 -> 82.85 | Hurt |
| Add the matching module to SPT initialization (ablation) | MoCo-v3 ViT-B/16 visual prompt tuning · VTAB-1k Structured | accuracy: 55.16 -> 56.51 | Helped |
| Add the orthogonalizing module to matching-only initialization (ablation) | MoCo-v3 ViT-B/16 visual prompt tuning · VTAB-1k Natural | accuracy: 76.50 -> 76.75 | Helped |
| Add the orthogonalizing module to matching-only initialization (ablation) | MoCo-v3 ViT-B/16 visual prompt tuning · VTAB-1k Specialized | accuracy: 82.85 -> 84.14 | Helped |
| Add the orthogonalizing module to matching-only initialization (ablation) | MoCo-v3 ViT-B/16 visual prompt tuning · VTAB-1k Structured | accuracy: 56.51 -> 56.68 | Helped |
| Use smaller k and larger λ (ablation) | VIPAMIN · Tasks with significant distribution shift | accuracy | Helped |
| Use larger k and smaller λ (ablation) | VIPAMIN · Tasks aligned with the pretraining distribution | accuracy | Helped |
Surprising findings
- Vanilla VPT with MAE backbone falls far below full fine-tuning on VTAB-1k (40.96 vs 61.28 mean), with structured tasks especially degraded.
- Prompt attention entropy stays near the maximum ln(N_e) for VPT across VTAB tasks, indicating prompts fail to specialize.
Limitations
- Hyperparameters k and lambda require empirical tuning with no principled selection rule.
- Evaluation is limited to image classification with ViT-B backbones in the main results; generalization to text or multimodal settings is untested.
- SPT/rand is used as a cheaper stand-in for SPT, and it underperforms SPT by about 2 points on MAE.
Experimental setup
Datasets: VTAB-1k CUB-200-2011 NABirds Stanford Dogs Stanford Cars Oxford Flowers ImageNet-1K
Benchmarks: VTAB-1k FGVC few-shot (CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, Stanford Cars)
Metrics: top-1 accuracy mean accuracy prompt attention entropy projection energy
Baselines: VPT SPT SPT/rand GatedPT E2VPT iVPT DA-VPT VFPT Full fine-tuning Linear Probing Adapter Bias Tuning
Builds on: Visual Prompt Tuning Self-Prompt Tuning MoCo-v3 MAE
Closest papers by meaning
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.92)
- Query Efficient Black-Box Visual Prompting with Subspace Learning (CVPR 2025 · similarity 0.92)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.90)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.89)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.