Research map / Vision-language and multimodal / CLIP and zero-shot
Consistency-guided Prompt Learning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoPrompt, a fine-tuning method for CLIP that enforces cosine consistency between trainable and frozen encoders on perturbed inputs (LLM-generated text and augmented images), combined with multi-modal prompts and adapters.
Problem
Few-shot fine-tuning of vision-language models overfits newly introduced parameters, degrading zero-shot generalization to novel classes and out-of-distribution data.
Approach
Enforce a cosine consistency loss between the trainable model's text and image embeddings and those of the frozen pre-trained CLIP, computed on perturbed inputs (GPT-generated descriptive sentences and augmented images), and combine this with joint multi-modal prompts and two-branch adapters so that extra parameters can be added without overfitting.</258core_idea>
Key results
- Base-to-novel average harmonic mean of 80.48% vs 78.55% for MaPLe and 79.97% for PromptSRC; novel accuracy 77.23%.
- Cross-dataset average accuracy 67.00% vs 66.30% for MaPLe when trained on ImageNet.
- Ablation: removing consistency constraint and perturbations drops HM to 78.45%, below the 78.55% of MaPLe, showing adapters+prompts overfit without consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CoPrompt instead of PromptSRC | CLIP ViT-B/16 · Base-to-novel generalization (11-dataset average) | HM: 79.97 -> 80.48 | Helped |
| use CoPrompt instead of MaPLe | CLIP ViT-B/16 · Cross-dataset evaluation (average) | Accuracy: 66.30 -> 67.00 | Helped |
| use CoPrompt instead of PromptSRC | CLIP ViT-B/16 · Domain generalization (average) | Accuracy: 60.65 -> 60.42 | Hurt |
| add adapters (ablation) | CoPrompt · Base-to-novel generalization | HM: 80.02 -> 80.48 | Helped |
| add input perturbations (ablation) | CoPrompt · Base-to-novel generalization | HM: 79.56 -> 80.48 | Helped |
| add the consistency constraint (ablation) | CoPrompt with adapters and prompts · Base-to-novel generalization | HM: 78.45 -> 80.48 | Helped |
| apply consistency to both image and text representations rather than image only (ablation) | CoPrompt · Base-to-novel generalization | Accuracy: 79.59 -> 80.48 | Helped |
| replace cosine consistency with MSE (ablation) | CoPrompt · Base-to-novel generalization | Accuracy: 80.48 -> 79.33 | Hurt |
| use descriptive LLM-generated text instead of the same text input (ablation) | CoPrompt · Base-to-novel generalization | Accuracy: 80.09 -> 80.48 | Helped |
| replace simple augmentation with hard image augmentation (ablation) | CoPrompt · Base-to-novel generalization | Accuracy: 80.48 -> 79.90 | Hurt |
| use adapters on both branches rather than the image branch only (ablation) | CoPrompt · Base-to-novel generalization | Accuracy: 80.10 -> 80.48 | Helped |
| use a 3-layer adapter instead of a 2-layer adapter (ablation) | CoPrompt · Base-to-novel generalization | Accuracy: 80.48 -> 79.75 | Hurt |
Surprising findings
- Adding adapters and prompts without the consistency constraint (78.45% HM) performs even worse than removing all components (78.55% for MaPLe baseline row), indicating overfitting.
- LLM-generated text prompts reduce zero-shot CLIP accuracy from 71.70% to 69.14% HM, though they help as consistency targets.
Limitations
- Roughly 2x FLOPs and about 25% more training time than MaPLe.
- Hard augmentations (RandAug) degrade performance, and the method is sensitive to lambda (performance drops at lambda=10).
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 EuroSAT DTD ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 EuroSAT DTD ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp Co-CoOp ProGrad KgCoOp MaPLe PromptSRC Bayesian Prompt UPT
Builds on: CoOp MaPLe PromptSRC KgCoOp CLIP-Adapter
Closest papers by meaning
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
- Prompt Learning via Meta-Regularization (CVPR 2024 · similarity 0.89)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.89)
- DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.