Research map / Vision-language and multimodal / CLIP and zero-shot
Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APP, a Bayesian prompt-learning method that uses an image-conditioned data-dependent prior and Stein Variational Gradient Descent (a Wasserstein gradient flow) to learn multiple context vectors for CLIP, improving few-shot, domain generalization and base-to-new performance.
Problem
Prompt learning for frozen vision-language models with MLE training overfits dominant image feature modes in few-shot settings and generalizes poorly under distribution shift, trading off seen and unseen performance.
Approach
Combines a data-dependent prior whose mean is an image-feature-conditioned mapping of averaged image features with particle-based posterior approximation via SVGD, so multiple context vectors cover multi-modal image features; at test time the prior is adapted with the test image via a weighted average of text features.
Key results
- APP reports the best average few-shot accuracy across 11 datasets (e.g., 64.67% at 1 shot and 75.81% at 16 shots), outperforming PLOT and ProDA.
- In base-to-new generalization, APP achieves the best harmonic mean on average (73.4 vs. 71.8 for ProDA) and on several datasets such as EuroSAT (H 63.1 vs. 55.2 for PLOT).
- On ImageNet domain generalization, APP improves source accuracy to 64.50% vs. 63.13% for CoCoOp and gains on ImageNetV2, Sketch, and A.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APP instead of ProDA | CLIP ResNet50 prompt learning · Few-shot classification average across 11 datasets | accuracy: 64.67 ± 0.26 vs 63.13 ± 0.26 (1 shot); 67.53 ± 0.50 vs 66.88 ± 0.22 (2 shots); 70.54 ± 0.15 vs 69.67 ± 0.16 (4 shots); 7 | Helped |
| Use APP instead of ProDA | CLIP ResNet50 prompt learning · Unseen classes average | H: 71.8 -> 73.4 | Helped |
| Use APP instead of the strongest listed baseline | CLIP ResNet50 prompt learning · ImageNet source and ImageNetV2, ImageNet-Sketch, ImageNet-R, ImageNet-A targets | accuracy: ImageNet: 63.13 ± 0.12 -> 64.50 ± 0.06; ImageNetV2: 54.97 ± 0.05 -> 57.10 ± 0.29; ImageNet-Sketch: 34.60 ± 0.22 -> 35.70 | Helped |
| Add a data-dependent prior to SGD prompt training (ablation) | APP · Caltech101, 1 shot | accuracy: 89.87 ± 0.02 -> 90.26 ± 0.21 | Helped |
| Add a data-dependent prior to SGD prompt training (ablation) | APP · EuroSAT, 1 shot | accuracy: 56.43 ± 2.31 -> 59.96 ± 4.77 | Helped |
| Add a data-dependent prior to SGD prompt training (ablation) | APP · Food101, 1 shot | accuracy: 78.53± 0.14 -> 78.81 ± 0.16 | Helped |
| Use SVGD instead of SGD, without a data-dependent prior (ablation) | APP · Caltech101, 1 shot | accuracy: 89.87 ± 0.02 -> 89.90 ± 0.07 | No clear effect |
| Use SVGD instead of SGD, without a data-dependent prior (ablation) | APP · EuroSAT, 1 shot | accuracy: 56.43 ± 2.31 -> 58.93 ± 4.28 | Helped |
| Use SVGD instead of SGD, without a data-dependent prior (ablation) | APP · Food101, 1 shot | accuracy: 78.53± 0.14 -> 78.51 ± 0.12 | No clear effect |
| Use 2 prompts instead of 4 (ablation) | APP · EuroSAT, 1 shot | accuracy: 60.04 ± 2.08 -> 52.06 ± 4.98 | Hurt |
| Vary the prompt count from 4 to 2 or 8 (ablation) | APP · Caltech101 and Food101, 1 shot | accuracy: Caltech101: 90.30 ± 0.33 (4) -> 89.19 ± 0.20 (2), 90.18 ± 0.28 (8); Food101: 78.87 ± 0.17 (4) -> 78.27± 0.16 (2), 78.90 | Mixed |
Surprising findings
- Input-conditioned CoCoOp loses to multi-prompt methods on base-to-new generalization for new classes in several datasets, showing a base/new trade-off.
- BPL fails with batch size 128 due to memory issues, while APP trains with the same batch size.
Limitations
- Gains over ProDA and PLOT are small on several datasets (e.g., Food101, Oxford Pets, FGVC-Aircraft at some shots), and some baselines win individual cells.
- The test-time adaptation uses a fixed weighted average with hyperparameter alpha chosen per setting rather than a principled posterior update.
- Experiments are limited to CLIP with ResNet50 as backbone.
Experimental setup
Datasets: Caltech101 DTD EuroSAT FGVCAircraft Oxford Flowers 102 OxfordPets Food101 StanfordCars SUN397 UCF101 ImageNet ImageNetV2
Benchmarks: Caltech101 DTD EuroSAT FGVCAircraft Oxford Flowers 102 OxfordPets Food101 StanfordCars SUN397 UCF101 ImageNet ImageNetV2
Metrics: accuracy harmonic mean variance of prompt counts per cluster
Baselines: CoOp CoCoOp PLOT ProDA BPL
Builds on: CoOp CoCoOp PLOT ProDA BPL Stein Variational Gradient Descent
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.93)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.91)
- Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models (ICML 2025 · similarity 0.91)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.