Research map / Vision-language and multimodal / CLIP and zero-shot
LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoR-VP, a visual prompt design that adds a low-rank product B·A to the resized full image, enabling shared row/column information and patch-specific adjustments with far fewer prompt parameters, paired with linear probing.
Problem
Existing visual prompting methods pad prompt parameters around the image, so only peripheral patches are modified, and per-patch prompts ignore shared inductive bias across patches.
Approach
Resize the image to full L×L and add a low-rank matrix product B·A (rank 4) as the visual prompt, so every pixel is modified and rows/columns share information, while using linear probing as the output transformation.
Key results
- LoR-VP averages a 3.2% gain over AutoVP and 2.1% over LP across ImageNet-1K and CLIP pre-trained model/dataset combinations, converging with 5x fewer epochs than AutoVP.
- On ResNet-18/CIFAR-10, LoR-VP reaches 88.64% vs 87.55% for AutoVP with 5K VP params vs 101K and 6x less training time (0.50h vs 2.61h).
- On OOD evaluation with Swin-B trained on ImageNet-1K, LoR-VP improves over AutoVP by an average of 10.6 points across ImageNet-R, Sketch, A and V2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace AutoVP with LoR-VP visual prompts | Swin-B prompted on ImageNet-1K · ImageNet-R | accuracy: 38.14 -> 52.27 | Helped |
| replace AutoVP with LoR-VP visual prompts | Swin-B prompted on ImageNet-1K · ImageNet-Sketch | accuracy: 28.89 -> 41.13 | Helped |
| replace AutoVP with LoR-VP visual prompts | Swin-B prompted on ImageNet-1K · ImageNet-A | accuracy: 17.91 -> 27.89 | Helped |
| replace AutoVP with LoR-VP visual prompts | Swin-B prompted on ImageNet-1K · ImageNet-V2 | accuracy: 67.38 -> 72.38 | Helped |
| replace AutoVP with LoR-VP visual prompts | ResNet-18 · CIFAR10 | accuracy: 87.55 -> 88.64 | Helped |
| replace AutoVP with LoR-VP visual prompts | ViT-B/16-P · Tiny-ImageNet | accuracy: 81.42 -> 84.40 | Helped |
| use LoR-VP visual prompting | YOLOv4 with ResNet-50 · Pascal VOC 2007 test | AP50: 73.07 -> 77.02 | Helped |
| use LoR-VP visual prompting | DeepLabv3+ with ResNet-50 · Pascal VOC 2012 validation | mIOU: 67.42 -> 68.55 | Helped |
| use LoR-VP prompts rather than LP alone | ViT-B/32 · Ten downstream datasets (average) | accuracy: 87.59 -> 88.94 | Helped |
| replace LP with FM output transformation (ablation) | LoR-VP on ViT-B/16-P · Tiny-ImageNet | accuracy: 84.40 -> 83.15 | Hurt |
| replace LP with FM output transformation (ablation) | LoR-VP on ViT-B/32 · CIFAR100 | accuracy: 88.65 -> 88.48 | Hurt |
| add the low-rank visual prompt while keeping output transformation (ablation) | LoR-VP on ViT-B/16-P · Tiny-ImageNet | accuracy: 82.75 -> 84.40 | Helped |
Surprising findings
- Patch-Same (one shared prompt tiled across all patches) outperformed the standard pad prompting baseline AutoVP in the preliminary study.
- LoR-VP combined with ILM needs a higher rank (around 16) to be optimal, unlike LP or FM where rank 4 suffices.
Limitations
- Mostly evaluated on image classification; detection and segmentation results are only in the appendix with one backbone.
- ILM-VP and high-resolution ImageNet-21K experiments were limited by GPU resources, so some baselines were run with reduced epochs.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K ImageNet-21K-P Tiny-ImageNet CIFAR-10 CIFAR-100 ImageNet-R ImageNet-Sketch ImageNet-A ImageNet-V2 Pascal VOC 2007 Pascal VOC 2012
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K ImageNet-R ImageNet-Sketch ImageNet-A ImageNet-V2 Pascal VOC 2007 Pascal VOC 2012
Metrics: top-1 accuracy AP AP50 AP75 mIOU training time epochs parameter count GPU memory inference latency
Baselines: AutoVP ILM-VP CLIP-VP LP VPT
Builds on: AutoVP ILM-VP CLIP-VP LoRA
Closest papers by meaning
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.91)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.90)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.89)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.