Research map / Vision-language and multimodal / CLIP and zero-shot
COMMA: Co-articulated Multi-Modal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COMMA, a multi-modal prompt learning method for CLIP that generates vision prompts from both branches' prompts and distills late-layer prompts toward hand-crafted CLIP embeddings to preserve generic knowledge.
Problem
Prompt learning for vision-language models like CLIP leaves vision and language prompts separated or uni-directionally coupled, and it degrades generalization to unseen classes relative to zero-shot CLIP.
Approach
Generate vision-branch prompts via attention that uses vision prompts as queries and text prompts as keys/values (co-articulated across both branches), and add a cosine-similarity loss in the last S layers pulling learned text prompts toward CLIP's hand-crafted prompt embeddings to retain generic knowledge.
Key results
- Base-to-novel average harmonic mean over 11 datasets: COMMA 79.04 vs MaPLe 78.55 and KgCoOp 77.00.
- Cross-dataset average over 10 target datasets: COMMA 66.84 vs MaPLe 66.30 and CoCoOp 65.74.
- Ablation: correlated prompts alone raise average accuracy from 75.86 to 78.02; knowledge transfer alone to 76.84; both to 79.04.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add correlated prompts only (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: +2.16% | Helped |
| add knowledge transfer only (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 0.98% | Helped |
| add correlated prompts and knowledge transfer (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: +3.18% | Helped |
| set knowledge-transfer weight λ to 0.5 instead of 1.0 (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 79.04 -> 78.68 | Hurt |
| set knowledge-transfer weight λ to 1.5 instead of 1.0 (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 79.04 -> 78.74 | Hurt |
| set prompt depth J to 3 instead of 9 (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 79.04 -> 77.98 | Hurt |
| set prompt depth J to 6 instead of 9 (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 79.04 -> 78.56 | Hurt |
| set prompt depth J to 12 instead of 9 (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 79.04 -> 78.74 | Hurt |
| increase prompt length P from 2 to 3 (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 79.04 -> 78.75 | Hurt |
| increase prompt length P from 2 to 4 (ablation) | COMMA · Base-to-novel, 11-dataset average | accuracy: 79.04 -> 78.54 | Hurt |
| use COMMA instead of MaPLe | CLIP ViT-B/16 · Cross-dataset, 10 target datasets average | accuracy: 66.30 -> 66.84 | Helped |
| use COMMA instead of MaPLe | CLIP ViT-B/16 · Domain generalization, ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R | accuracy | Helped |
Surprising findings
- Prior prompt-learning methods such as CoOp, CoCoOp and KgCoOp achieve higher base accuracy than CLIP but lower novel-class accuracy, indicating forgetting of generic knowledge.
Limitations
- Gains over MaPLe are modest (e.g., +0.49 HM on base-to-novel average).
- Parameter count is higher than CoOp and KgCoOp (4.87M vs 2048).
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean FPS parameter count
Baselines: CLIP CoOp CoCoOp KgCoOp MaPLe ProGrad
Builds on: CLIP CoOp MaPLe CoCoOp KgCoOp
Closest papers by meaning
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.90)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.90)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.89)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.