Research map / Vision-language and multimodal / CLIP and zero-shot
LaViP: Language-Grounded Visual Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaViP, a language-grounded, input-aware visual prompting method that adapts CLIP's visual encoder via low-rank, text-conditioned padding prompts without modifying model weights, and extends to base-to-novel generalization and black-box (SPSA) settings.
Problem
Visual prompting for vision-language models treats class semantics and visual encoding separately, learns input-agnostic prompts, converges slowly, and generalizes poorly to unseen classes; adaptation is also hard when model parameters are inaccessible.
Approach
Generate padding-style visual prompts as a low-rank product of a text-derived class-semantic matrix (from frozen CLIP text encoder) and an image-dependent matrix (from frozen CLIP image features with feature-wise scale/shift), and extend to novel classes via a Kronecker-product similarity to base classes without retraining.
Key results
- Few-shot (16-shot, ViT-B/16) average accuracy 74.59% vs VP 71.26% and CLIP zero-shot 62.75%, winning 10 of 12 datasets.
- Base-to-novel: LaViP harmonic mean average 78.31% vs CoCoOp 75.67%, while trailing MaPLe (78.75%) by 0.44 points.
- Black-box BlackLaViP reaches 65.77% average vs BlackVIP 67.47% (about 97.5% of BlackVIP) with claimed over 15x faster convergence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace fixed pixel-space prompts with language-grounded, input-specific prompts | LaViP vs. VP · Few-shot transfer, 12 datasets | Avg. accuracy: 71.26 -> 74.59 | Helped |
| Use language-grounded visual prompts instead of VP | LaViP vs. VP · Full-dataset transfer, 9 datasets | Avg. accuracy: 75.72 -> 80.99 | Helped |
| Use LaViP rather than MaPLe | Base-to-novel generalization · 10 datasets | Avg. HM: 78.75 -> 78.31 | Hurt |
| Use LaViP rather than CLIP zero-shot | Base-to-novel generalization · 10 datasets, novel classes | Avg. Novel accuracy: 74.83% to 75.70% | Helped |
| Use custom context-specific prompt templates instead of the same base template (ablation) | LaViP · Few-shot transfer, 6 datasets | Avg. accuracy: 86.05 -> 86.47 | Helped |
| Set prompt rank r=64 instead of the reported LaViP setting (ablation) | LaViP · Five-dataset rank ablation | Avg. accuracy: 87.48 -> 87.36 | No clear effect |
| Set prompt rank r=32 instead of the reported LaViP setting (ablation) | LaViP · Five-dataset rank ablation | Avg. accuracy: 87.48 -> 86.77 | No clear effect |
| Set prompt rank r=96 instead of the reported LaViP setting (ablation) | LaViP · Five-dataset rank ablation | Avg. accuracy: 87.48 -> 86.77 | No clear effect |
| Use SPSA gradient-free optimization (ablation) | BlackLaViP vs. BlackVIP · Black-box few-shot transfer, 10 datasets | Avg. accuracy: 67.47 -> 65.77 | Hurt |
Surprising findings
- LaViP underperforms zero-shot CLIP on low-resolution and less semantically varied datasets such as SVHN, CIFAR100 and PCam.
- Average novel-class accuracy of LaViP is only marginally above CLIP zero-shot (75.70 vs 74.83), and it improves novel accuracy on just 2 of 10 datasets.
Limitations
- Underperforms on low-resolution datasets (SVHN, CIFAR10/100) and datasets with low semantic variation (PCam).
- Relies on a single hand-crafted prompt template; performance depends on context tokens in the template.
- Gains over MaPLe and CoCoOp on base-to-novel are small or negative on several datasets; ImageNet cross-dataset result is suboptimal.
Experimental setup
Datasets: Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT RESISC45 CLEVR
Benchmarks: Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT RESISC45 CLEVR
Metrics: accuracy harmonic mean
Baselines: CLIP zero-shot VP CoOp CoCoOp MaPLe BlackVIP BAR CLIP linear probe
Builds on: Visual Prompting (VP) BlackVIP CLIP CoOp CoCoOp MaPLe SPSA
Closest papers by meaning
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.91)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.91)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.91)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.90)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.