Research map / Label-efficient and robust learning / Recognition and long-tail
CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoCoA-Mix, a prompt-tuning method for CLIP that pairs a confusion-aware loss (CoA-loss) for specialization with confidence-aware mixture weights (CoA-weights) for generalization, without adding network parameters.
Problem
Prompt tuning of vision-language models trades off specialization on the training task against generalization to unseen classes and domains, and cross-entropy handles confusing class pairs poorly when the visual encoder is frozen.
Approach
Adds a loss term (1 - p(y)) that increases gradients for confusing samples near the decision boundary, and mixes per-prompt predictions with weights tuned separately for in-class (cross-entropy) and out-class (entropy margin) domains so that specialization and generalization improve together.
Key results
- Base-to-new average harmonic mean of 77.03 vs. 66.82 for zero-shot CLIP (reported in the table; the abstract's claimed 15.28% gain is replaced by ? in the alternative version).
- Cross-dataset transfer: target average 65.27 and harmonic mean 66.07, the highest among compared methods.
- FSCIL on CIFAR100 reaches mean accuracy 83.5 with PD 7.4, versus 81.4 for FSPT-FSCIL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CoA-loss and confidence-aware prompt weights | CLIP prompt tuning · Base-to-new generalization (11-dataset average) | harmonic mean: 76.38 -> 77.03 | Helped |
| add CoA-loss and confidence-aware prompt weights | CLIP prompt tuning · CIFAR100 FSCIL | mean accuracy: 81.4 -> 83.5 | Helped |
| add CoA-loss and confidence-aware prompt weights | CLIP prompt tuning · Cross-dataset transfer | harmonic mean: 65.26 -> 66.07 | Helped |
| add CoA-loss to cross-entropy prompt training (ablation) | CLIP prompt tuning · Base-to-new generalization (11-dataset average, Base) | accuracy: 65.1 -> 78.6 | Helped |
| add a uniform prompt ensemble to CoA-loss (ablation) | CoA-loss prompt tuning · Base-to-new generalization (11-dataset average, New) | accuracy: 68.5 -> 73.7 | Helped |
| replace uniform ensemble weights with CoA-weights (ablation) | CoA-loss prompt ensemble · Base-to-new generalization (11-dataset average, New) | accuracy: 73.7 -> 75.1 | Helped |
| use two-stage rather than one-stage CoA-weight optimization (ablation) | CoCoA-Mix · Base-to-new generalization (11-dataset average) | harmonic mean: 77.0 -> 77.0 | No clear effect |
Surprising findings
- Naive uniform ensembling of CLIP prompts with no tuning already raises harmonic mean from 66.8 to 72.3, showing that mixing alone helps generalization.
- CoA-loss alone improves base accuracy but yields limited new-class generalization, so the mixture weights are needed for the trade-off.
Limitations
- Out-class sets are generated from random words via an API, which adds an external dependency to the method.
- Generalization gains in cross-dataset transfer are smaller than in base-to-new generalization, as the paper itself notes.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 CIFAR100
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 CIFAR100 FSCIL
Metrics: accuracy harmonic mean mean accuracy over sessions performance difference
Baselines: CLIP CoOp ProGrad KgCoOp MaPLe DePT L2P CoOp-FSCIL FACT w/ CLIP FSPT-FSCIL CoCoOp
Builds on: CoOp CLIP KgCoOp ProGrad MaPLe DePT
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.88)
- DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models (CVPR 2025 · similarity 0.88)
- ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models (CVPR 2024 · similarity 0.87)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.87)
- Consistency-guided Prompt Learning for Vision-Language Models (ICLR 2024 · similarity 0.87)
- Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.