Research map / Vision-language and multimodal / CLIP and zero-shot
Learning to Prompt with Text Only Supervision for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns deep language prompts for CLIP using only GPT-3-generated text descriptions via a contextual MSE mapping objective, enabling zero-shot transfer of prompts to new classes and datasets without any images.
Problem
Adapting CLIP for downstream tasks usually needs labeled images for prompt learning, which overfits and is impractical, while LLM-based prompt ensembling is training-free and its class-specific descriptions do not transfer to new classes or datasets.
Approach
Learn deep text prompts with text-only supervision by training them so that the prompted class-name template feature maps (MSE) to the frozen CLIP feature of LLM-generated descriptions of that class, so the learned prompts carry LLM context and transfer to unseen classes and datasets.
Key results
- Cross-dataset transfer: ProText averages 67.23% over 10 target datasets versus 65.15% for CLIP/CuPL and surpasses image-supervised MaPLe (66.30%) on 9/10 datasets.
- Base-to-novel: ProText novel-class average 76.98% versus CuPL 74.22%, with HM 74.91% vs 73.38%.
- ImageNet with CuPL-equivalent text data: ProText 70.22% vs CuPL 69.62%, the best among text-only ensembling methods in that ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn prompts from text-only CuPL descriptions instead of using CuPL prompt ensembling | CLIP · ImageNet | top-1 accuracy: 69.62 -> 70.22 | Helped |
| Train prompts on ImageNet text-only LLM data and transfer to target datasets | CLIP · Cross-dataset transfer average | accuracy: +2.1% | Helped |
| Train prompts on ImageNet text-only data | CLIP · Domain generalization average | accuracy: +0.44% | Helped |
| Train on base-class LLM descriptions and transfer learned prompts to novel classes | CLIP · Novel classes average | accuracy: 2.76% | Helped |
| Use MSE rather than contrastive contextual-mapping loss (ablation) | ProText · ImageNet | top-1 accuracy: 68.12 -> 70.22 | Helped |
| Use MSE rather than L1 contextual-mapping loss (ablation) | ProText · ImageNet | top-1 accuracy: 69.96 -> 70.22 | Helped |
| Replace GPT-3 text data with Alpaca text data (ablation) | ProText · ImageNet | top-1 accuracy: 70.22 -> 67.10 | Hurt |
| Replace prompt learning with a linear adaptor (ablation) | ProText · ImageNet | top-1 accuracy: 70.22 -> 69.36 | Hurt |
| Replace prompt learning with an MLP adaptor (ablation) | ProText · ImageNet | top-1 accuracy: 70.22 -> 69.24 | Hurt |
| Increase the number of LLM descriptions per class (ablation) | ProText · ImageNet | top-1 accuracy | Helped |
| Ensemble descriptions into one feature per class instead of training on individual descriptions (ablation) | ProText · ImageNet | top-1 accuracy | Hurt |
| Set prompt length to 16 (ablation) | ProText · ImageNet | top-1 accuracy | Helped |
Surprising findings
- CoOp with 16-shot labeled ImageNet reduces average CLIP cross-dataset performance by 1.27%, suggesting image-supervised prompt learning overfits to source data.
- Adapter-based contextual mapping performed worse than prompt learning, with the authors conjecturing adapters lose CLIP's generalization.
Limitations
- Performance depends on LLM text data quality and size; per-dataset training configurations are tuned on validation splits.
- Gains over CuPL in the supervised text-only setting are modest (+0.59% average) and it underperforms CuPL on EuroSAT and Aircraft-style cases such as EuroSAT (-1.74%).
Experimental setup
Datasets: ImageNet-1k LLM text data (GPT-3 via CuPL queries) CuPL LLM data ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Metrics: top-1 accuracy base accuracy novel accuracy harmonic mean average accuracy
Baselines: CLIP CuPL DCLIP WaffleCLIP CoOp CoCoOp MaPLe PromptSRC Linear Probe
Builds on: CoOp CuPL Deep Language Prompting CLIP
Closest papers by meaning
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.93)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.93)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.93)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.92)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.92)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.