Research map / Vision-language and multimodal / CLIP and zero-shot
Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes test-time dynamic fusion of zero-shot CLIP and few-shot prompt-tuned classifiers, weighted by a competition-based score derived from OOD detectors' in-distribution scores, to improve base-to-novel generalization of CoOp, CoCoOp, ProGrad and KgCoOp.
Problem
Vision-language prompt tuning methods such as CoOp fit base classes well but lose accuracy on novel classes; balancing base and novel performance is hard.
Approach
Treat the choice between zero-shot and few-shot classifiers as an OOD detection problem: a sigmoid over the difference of ID scores from the two classifiers (MSP, MaxLogit, Energy or entropy) gives an instance-wise weight to fuse the classifiers at test time without retraining.
Key results
- With OOD detection, CoOp's harmonic mean over 11 datasets rises by 2.6 points (74.24 to 76.85), and ProGrad's by 1.5 points (75.70 to 77.20).
- Pro+Kg (ProGrad fused with KgCoOp) reaches 77.60 harmonic mean on the 11-dataset average, and 60.36% average target accuracy in domain generalization.
- Dynamic weighting with alpha=64 (76.85 H) beats the best static weight (75.81 H at 0.50) for CoOp+.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OOD-based dynamic fusion to CoOp | CoOp · Base-to-novel (11 datasets) | H: 74.24 -> 76.85 | Helped |
| Add OOD-based dynamic fusion to ProGrad | ProGrad · Base-to-novel (11 datasets) | H: 75.70 -> 77.20 | Helped |
| Add OOD-based dynamic fusion to CoCoOp | CoCoOp · Base-to-novel (11 datasets) | H: 76.04 -> 76.74 | Helped |
| Add OOD-based dynamic fusion to KgCoOp | KgCoOp · Base-to-novel (11 datasets) | H: 76.97 -> 76.94 | No clear effect |
| Replace the zero-shot classifier with ProGrad and fuse with KgCoOp | KgCoOp+ · Base-to-novel (11 datasets) | H: 77.20 -> 77.60 | Helped |
| Replace the zero-shot classifier with ProGrad and fuse with KgCoOp | KgCoOp+ · Domain generalization target domains (average) | accuracy: 59.88 -> 60.36 | Helped |
| Use dynamic weighting instead of a fixed fusion weight of 0.50 (ablation) | CoOp+ · Base-to-novel (11 datasets) | H: 75.81 -> 76.85 | Helped |
| Use competition-based MSP scoring rather than MSP from either classifier alone (ablation) | CoOp+ · Base-to-novel (11 datasets) | H: 76.09 -> 77.15; 76.43 -> 77.15 | Helped |
| Use MaxLogit instead of Entropy as the OOD detector (ablation) | CoOp+ · Base-to-novel (11 datasets) | H: 76.85 -> 75.86 | Hurt |
| Use Energy instead of Entropy as the OOD detector (ablation) | CoOp+ · Base-to-novel (11 datasets) | H: 76.85 -> 75.85 | Hurt |
| Use binary weights (α=∞) instead of continuous weights (ablation) | CoOp+ with MSP detector · Base-to-novel (11 datasets) | H: 77.15 -> 77.13 | Hurt |
| Increase the scaling factor α (ablation) | CoOp+ · Base-to-novel (11 datasets) | Base and Novel accuracy | Mixed |
Surprising findings
- Weak OOD detectors (about 50% accuracy) still improve harmonic mean, because the fused classifier outperforms either classifier alone.
- MaxLogit and Energy scores do not work well for this fusion, contrary to their strong performance in standard OOD detection.
Limitations
- Effectiveness is reduced when the two fused classifiers have little complementarity (e.g., KgCoOp gains little).
- Domain generalization gains are small because the few-shot classifier dominates on shared classes.
Experimental setup
Datasets: ImageNet Caltech101 Food101 StanfordCars OxfordPets Flowers102 FGVCAircraft UCF101 EuroSAT DTD SUN397 ImageNetV2
Benchmarks: ImageNet Caltech101 Food101 StanfordCars OxfordPets Flowers102 FGVCAircraft UCF101 EuroSAT DTD SUN397 ImageNetV2
Metrics: base accuracy novel accuracy harmonic mean target domain accuracy
Baselines: CLIP CoOp CoCoOp ProGrad KgCoOp
Builds on: CoOp CoCoOp ProGrad KgCoOp CLIP maximum softmax probability MaxLogit energy-based OOD detection
Closest papers by meaning
- DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection (ICML 2024 · similarity 0.90)
- Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.89)
- Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models (ICML 2024 · similarity 0.89)
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.88)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.88)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.