Research map / Vision-language and multimodal / CLIP and zero-shot
Concept-Guided Prompt Learning for Generalization in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Concept-Guided Prompt Learning (CPL), which builds a visual concept cache from CLIP and uses a transformer-decoder projector over multi-level visual features to refine text features for better generalization of CLIP.
Problem
Prompt-tuning methods for CLIP (CoOp, CoCoOp) generalize poorly on fine-grained datasets because they only use global class-specific features and ignore transferable low-level visual concepts such as colors, shapes, and materials.
Approach
Mine a visual concept cache by matching ~2000 text concept descriptions to training image features with frozen CLIP, retrieve top-K concepts as concept-guided prompts, and use a transformer-decoder projector that maps multi-level visual features (globally pooled per layer) into text features, combined with a learnable residual task adapter.
Key results
- Average base-to-novel harmonic mean over 11 datasets of 81.08 (ViT-B/16), vs. 78.55 for MaPLe; novel-class average 78.03.
- Cross-dataset transfer from ImageNet: target average 68.07% vs. 66.30% for MaPLe.
- 16-shot ImageNet accuracy 66.92% with 50 minutes of training vs. CoOp at 14h 40m for 62.26%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add concept-guided prompting (ablation) | CPL on ImageNet, 16-shot · ImageNet, 16-shot | accuracy: 60.33 -> 64.38 | Helped |
| Add the projector after concept-guided prompting (ablation) | CPL on ImageNet, 16-shot · ImageNet, 16-shot | accuracy: 64.38 -> 66.35 | Helped |
| Add the task adapter to concept-guided prompting and projector (ablation) | CPL on ImageNet, 16-shot · ImageNet, 16-shot | accuracy: 66.35 -> 66.92 | Helped |
| Set selected concept count K to 6 instead of 10 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 65.33 | Hurt |
| Set selected concept count K to 8 instead of 10 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 66.28 | Hurt |
| Set selected concept count K to 12 instead of 10 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 66.56 | Hurt |
| Set selected concept count K to 14 instead of 10 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 66.31 | Hurt |
| Set text concept set size I to 1000 instead of 2000 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 63.67 | Hurt |
| Set text concept set size I to 1500 instead of 2000 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 65.88 | Hurt |
| Set text concept set size I to 2500 instead of 2000 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 66.71 | Hurt |
| Set text concept set size I to 3000 instead of 2000 (ablation) | CPL · ImageNet, 16-shot | accuracy: 66.92 -> 66.23 | Hurt |
| Use CPL instead of MaPLe | Base-to-novel generalization, 11-dataset average · Base-to-novel generalization, 11-dataset average (novel classes) | accuracy: +2.89 | Helped |
Surprising findings
- Linear probe CLIP underperforms zero-shot CLIP on 16-shot ImageNet (56.13 vs. 60.33).
- Increasing the concept dictionary size beyond 2000 decreases accuracy, attributed to different text concepts matching the same visual concept.
Limitations
- Concept dictionary of 2000 descriptions is fixed and hand-gathered from existing attribute datasets; larger dictionaries hurt performance.
- Gains on some datasets are small (e.g., +0.11 base on EuroSAT, +0.33 base on Caltech101).
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNet-V2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNet-V2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp MaPLe ProGrad KgCoOp PLOT DeFo TPT TaskRes
Builds on: CLIP CoOp CoCoOp Transformer decoder
Closest papers by meaning
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.91)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.91)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.90)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.90)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.