Research map / Vision-language and multimodal / CLIP and zero-shot
DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces open-world prompt tuning (OPT), where prompts are tuned on base classes and evaluated on mixed base and new classes, and proposes DeCoOp, which combines OOD detection (new-class detectors) with base and new sub-classifiers built on CLIP prompt tuning.
Problem
Few-shot prompt tuning of VLMs like CLIP is evaluated separately on base and new classes, but real applications face a mix of base and new classes at test time without knowing which group a sample belongs to, and learned prompts can degrade new-class discriminability.
Approach
Decompose the classification into an OOD detection step (deciding base vs. new class space) and a classifier step; use the zero-shot CLIP classifier for samples detected as new and a prompt-tuned sub-classifier for samples detected as base. New-class detectors are trained with a leave-out class partition ensemble (K=3) using an entropy-margin loss, and thresholds are set via Otsu's method.
Key results
- DeCoOp averages 69.69% accuracy and 76.13 H with ViT-B/16 across 11 datasets, versus 65.57% accuracy for CoOp and 67.67% for CoCoOp.
- On base-to-new AUROC with MSP, DeCoOp averages 91.57 versus 80.67 for CLIP and 80.71 for SHIP (ViT-B/16).
- DePt framework alone raises average accuracy from 65.39 (Prompt Ens.) to 68.03 with ViT-B/16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate OOD detection with prompt tuning (DePt) | CLIP zero-shot and CoOp prompt tuning · 11-dataset average, ViT-B/16 | Accuracy: 63.92 -> 68.03 | Helped |
| Integrate OOD detection with prompt tuning (DePt) | CLIP zero-shot and CoOp prompt tuning · 11-dataset average, ViT-B/16 | New Acc.: 65.49 -> 68.15 | Helped |
| Add DeCoOp new-class detectors and sub-classifiers (ablation) | DePt · 11-dataset average, ViT-B/16 | Accuracy: 68.03 -> 69.69 | Helped |
| Add DeCoOp new-class detectors and sub-classifiers (ablation) | DePt · 11-dataset average, ViT-B/32 | Accuracy: 62.92 -> 65.75 | Helped |
| Use DeCoOp instead of comparison methods | Prompt-based classification · 11-dataset average, ViT-B/16 | Accuracy: 67.67 -> 69.69 | Helped |
| Use DeCoOp new-class detectors | CLIP, Prompt Ens., CoOp, CoCoOp, and Ship · 11-dataset average, ViT-B/16 | AUROC: 81.65 -> 91.57 | Helped |
| Combine multiple CoOp prompts rather than use a single CoOp prompt (ablation) | CoOp · Flowers102 | Accuracy: 72.11 -> 71.62; 72.11 -> 73.12; 72.11 -> 71.89 | Mixed |
| Combine multiple CoOp prompts rather than use a single CoOp prompt (ablation) | CoOp · DTD | Accuracy: 48.18 -> 50.08; 48.18 -> 49.99; 48.18 -> 49.69 | Helped |
| Combine multiple CoOp prompts rather than use a single CoOp prompt (ablation) | CoOp · Caltech101 | Accuracy: 93.24 -> 93.31; 93.24 -> 92.89; 93.24 -> 92.87 | Mixed |
| Combine multiple CoOp prompts rather than use a single CoOp prompt (ablation) | CoOp · StanfordCars | Accuracy: 63.81 -> 64.20; 63.81 -> 64.32; 63.81 -> 65.03 | Helped |
Surprising findings
- The H metric does not reliably track accuracy; on FGVCAircraft higher H did not yield higher accuracy.
- CoOp prompt ensembling (2, 4, or 6 CoOp prompts) does not consistently improve over single CoOp, sometimes degrading results.
Limitations
- Two-stage inference roughly doubles running time compared to CoOp and requires K times more detector computation.
- Only an initial step in combining OOD detection with prompt tuning; the DePt zero-shot/MSP detector choice is simple.
- Hyperparameter margin gamma and K are fixed heuristically.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Metrics: Accuracy H (harmonic mean of base and new accuracy) AUROC
Baselines: CLIP Prompt Ensemble CoOp CoCoOp SHIP DePt
Builds on: CoOp CoCoOp CLIP MSP SHIP
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning (ICML 2025 · similarity 0.90)
- Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning (AAAI 2024 · similarity 0.90)
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.89)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.89)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.