Research map / Vision-language and multimodal / CLIP and zero-shot
VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VaMP, a variational multi-modal prompt learning framework that generates instance-conditioned, uncertainty-aware text prompts sampled from a learned posterior with a class-prototype-based prior, evaluated on CLIP-based few-shot, base-to-novel, cross-dataset and domain generalization benchmarks.
Problem
Adapting vision-language models like CLIP to downstream tasks with limited supervision; existing multi-modal prompt methods use fixed, shared, deterministic prompts that cannot capture instance-level variation or uncertainty.
Approach
Treat layer-wise text prompts as latent variables with an image-conditioned Gaussian posterior and a class-prototype-conditioned prior, train with an ELBO via reparameterized sampling, and average predictions over S samples at inference, building on MMRL/MaPLe-style multi-modal prompting.
Key results
- Base-to-novel average over 11 datasets: 86.45 base, 78.67 novel, 82.37 harmonic mean, versus 81.20 HM for MMRL.
- Domain generalization average target accuracy 61.73%, reported as outperforming MMRL by 1.20%, with ImageNet source accuracy 72.83%.
- Cross-dataset average over 10 targets reaches 67.74%, 0.49 points above MMRL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VaMP to MMRL | MMRL with CLIP ViT-B/16 · Base-to-novel generalization (11-dataset average) | H: 81.20 -> 82.37 | Helped |
| add VaMP to MMRL | MMRL trained on ImageNet · Cross-dataset generalization (10-dataset average) | accuracy: 67.25 -> 67.74 | Helped |
| add VaMP to MMRL | MMRL trained on ImageNet · Domain generalization (ImageNet-Sketch) | accuracy: 49.17 -> 49.69 | Helped |
| replace task-specific prompts with sample-specific prompts (ablation) | MaPLe and MMRL · Base-to-novel generalization (11-dataset average) | novel accuracy: 75.14 -> 76.95; 77.16 -> 78.13 | Helped |
| replace deterministic prompts with variational prompts (ablation) | MaPLe and MMRL · Base-to-novel generalization (11-dataset average) | novel accuracy: 76.95 -> 77.32; 78.13 -> 78.45 | Helped |
| replace normal Gaussian prior with class-aware prior (ablation) | MaPLe and MMRL · Base-to-novel generalization (11-dataset average) | novel accuracy: 77.32 -> 78.07; 78.45 -> 78.67 | Helped |
| increase inference samples from S=1 to S=10 (ablation) | MMRL + VaMP · Base-to-novel generalization (11-dataset average) | H: 81.44 -> 82.37 | Helped |
| increase inference samples from S=10 to S=50 (ablation) | MMRL + VaMP · Base-to-novel generalization (11-dataset average) | H: 82.37 -> 82.01 | Hurt |
| set prompt depth to J=6, H=7 rather than J=4, H=3 (ablation) | MMRL + VaMP · Base-to-novel generalization (11-dataset average) | H: 81.13 -> 82.37 | Helped |
| increase prompt token length from M=1 to M=5 (ablation) | MMRL + VaMP · Base-to-novel generalization (11-dataset average) | H: 81.30 -> 82.37 | Helped |
| increase prompt token length from M=5 to M=8 (ablation) | MMRL + VaMP · Base-to-novel generalization (11-dataset average) | H: 82.37 -> 82.00 | Hurt |
Surprising findings
- Reported ablation shows the variational prompt component adds only about 0.26 HM on MMRL (82.10 vs 81.84 in the MMRL ablation), and the class-aware prior adds 0.27 HM, suggesting modest per-component gains.
Limitations
- Class-aware prior requires class prototypes from labeled training data, so it does not apply in zero-label settings.
- Experiments are limited to classification; generative or structured tasks such as captioning or VQA are not evaluated.
- Vision-side prompts are shared and deterministic; only text-side prompts are variational.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp ProDA KgCoOp MaPLe PromptSRC TCP MMA 2SFS SkipT MMRL
Builds on: CLIP MaPLe MMRL CoCoOp Bayesian Prompt Learning
Closest papers by meaning
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.88)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.88)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.