Research map / Vision-language and multimodal / CLIP and zero-shot
CoPL: Contextual Prompt Learning for Vision-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoPL, a prompt-learning method for CLIP that conditions learnable prompt tokens on local ViT patch features and weights them via attention, improving base-to-novel and domain generalization over CoCoOp and related methods.
Problem
Existing CLIP prompt learning methods such as CoCoOp condition prompts on global image features and weight all prompt tokens equally, which hurts generalization to unseen classes and out-of-distribution data.
Approach
Align learnable prompt tokens to local (patch-level) image features using an additive-style attention (Luong/Bahdanau-like content score) so each prompt receives image-dependent importance weights, producing contextual prompts while keeping CLIP frozen.
Key results
- Average unseen-class accuracy of 75.6% across 11 datasets versus 71.7% for CoCoOp and 74.2% for CLIP; average harmonic mean 78.5% vs 75.8% for CoCoOp.
- Domain generalization from ImageNet: average 61.0% across ImageNetV2, Sketch, A, R, above CoPrompt (60.4%) and CoCoOp (59.9%).
- Cross-dataset zero-shot (trained on Caltech101): average 65.1% vs 63.7% for CoCoOp; but CoPL is lower on EuroSAT unseen (34.2 vs 60.0 in base-to-novel table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace global image features with local image features for prompt learning (ablation) | CoPL · Caltech101, Seen | accuracy: 97.2 -> 98.1 | Helped |
| replace global image features with local image features for prompt learning (ablation) | CoPL · Caltech101, Unseen | accuracy: 94.1 -> 94.9 | Helped |
| replace global image features with local image features for prompt learning (ablation) | CoPL · DTD, Seen | accuracy: 77.4 -> 78.0 | Helped |
| replace global image features with local image features for prompt learning (ablation) | CoPL · DTD, Unseen | accuracy: 39.9 -> 50.0 | Helped |
| use contextual local features and prompt weighting instead of CoCoOp conditioning | CoPL · 11-dataset classification, Unseen (16-shot/class) | accuracy: +3.9% | Helped |
| use contextual local features and prompt weighting instead of CoCoOp conditioning | CoPL · Cross-dataset zero-shot, 9 target datasets | accuracy: 63.7 -> 65.1 | Helped |
| use contextual local features and prompt weighting instead of CoPrompt | CoPL · Domain generalization, ImageNet source; 4 targets | accuracy: 60.4 -> 61.0 | Helped |
| use contextual local features and prompt weighting instead of CoCoOp | CoPL · Incremental test, average over 11 datasets | accuracy: 69.1 -> 74.7 | Helped |
| use contextual local features and prompt weighting instead of CoCoOp | CoPL · EuroSAT, Unseen (16-shot/class) | accuracy: 60.0 -> 34.2 | Hurt |
Surprising findings
- CoPL underperforms CoCoOp on EuroSAT unseen classes, attributed to satellite images lacking salient local objects.
- Plain CoOp unseen accuracy (63.2%) falls below zero-shot CLIP (74.2%), showing learned static prompts overfit base classes.
Limitations
- Underperforms CoCoOp on EuroSAT unseen classes because images lack salient local regions.
- Relies on local patch features, which may be less effective for scene-like or texture images.
- Reported results on several tables are borrowed from prior work and some table numbers appear inconsistent with text claims.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Benchmarks: Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp KgCoOp ProGrad MaPLe CoPrompt
Builds on: CoOp CoCoOp CLIP KgCoOp
Closest papers by meaning
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.92)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.91)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.91)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.