Research map / Vision-language and multimodal / CLIP and zero-shot
Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that prompt tuning of CLIP causes a base-vs-new calibration trade-off (CoOp overconfident on new classes, KgCoOp underconfident on base) and proposes Dynamic Outlier Regularization (DOR), which regularizes textual features of WordNet-derived outlier nouns to match zero-shot CLIP.
Problem
Confidence calibration of CLIP after prompt tuning degrades on base or new classes; existing prompt tuning methods trade calibration between base and new classes.
Approach
Attribute new-class overconfidence to CE-driven growth of textual feature divergence, and regularize only textual outlier nouns (WordNet words near but not overlapping base classes, resampled each epoch) toward zero-shot CLIP features, leaving base-class features unconstrained.
Key results
- DOR reduces CoOp's average new-class ECE from 14.58% to 6.49% while base ECE moves from 3.07% to 2.67% (11 datasets, ViT-B/16).
- On base-to-new accuracy, DOR raises CoOp's new-class accuracy from 61.74% to 72.01% and CoCoOp/MaPLe new accuracy beyond zero-shot CLIP (74.59 and 75.89 vs 74.32 for ZSCLIP).
- On domain generalization, DOR reduces CoOp's average target ECE from 7.18% to 4.89%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DOR to CoOp | CoOp with CLIP ViT-B/16 · 11 base-to-new datasets (New classes) | ECE (%): 14.58 -> 6.49 | Helped |
| Add DOR to CoOp | CoOp with CLIP ViT-B/16 · 11 base-to-new datasets (Base classes) | ECE (%): 3.07 -> 2.67 | Helped |
| Add DOR to prompt-tuning methods | CoCoOp and MaPLe · 11 base-to-new datasets (New classes) | ECE (%): 6.14 -> 4.02; 5.46 -> 4.44 | Helped |
| Add DOR to regularization-based prompt-tuning methods | KgCoOp, TCP, PromptSRC, CoPrompt, and PromptKD · 11 base-to-new datasets (HM) | ECE (%): 5.15 -> 5.03; 4.39 -> 4.29; 3.95 -> 3.84; 4.26 -> 3.83; 4.56 -> 4.24 | Helped |
| Add DOR to CoOp | CoOp with CLIP ViT-B/16 · 11 base-to-new datasets (HM) | Accuracy (%): 72.36 -> 77.61 | Helped |
| Add DOR to CoOp | CoOp with CLIP ViT-B/16 · ImageNet-1k source, ImageNet target datasets (Target AVG) | ECE (%): 7.18 -> 4.89 | Helped |
| Add DOR to CoOp | CoOp with CLIP ViT-B/16 · ImageNet-1k source, ImageNet target datasets (Target AVG) | Accuracy (%): 57.72 -> 59.73 | Helped |
| Add visual outlier regularization | VPT with CLIP ViT-B/16 · Flowers102, Cars, DTD, and UCF101 (New classes average) | ECE (%): 6.99 -> 5.58 | Helped |
| Exclude all new classes from the DOR outlier pool (ablation) | CoOp + DOR · 11 base-to-new datasets (New classes) | ECE (%): 6.49 -> 6.77 | Hurt |
| Use far-OOD instead of near-OOD outliers (ablation) | CoOp + DOR · 11 base-to-new datasets | ECE (%): 4.89 vs 5.34 | Hurt |
| Use random-OOD instead of near-OOD outliers (ablation) | CoOp + DOR · 11 base-to-new datasets | ECE (%): 4.89 vs 5.07 | Hurt |
| Use oracle new-class outliers (ablation) | CoOp + DOR · 11 base-to-new datasets (New classes) | ECE (%): 7.09 vs 4.34 | Helped |
Surprising findings
- Random selection of outlier words serves as a strong baseline, nearly matching the near-OOD selection.
- Regularizing with base-class texts (KgCoOp) reduces gradient alignment with CE, whereas outlier regularization shows far fewer conflicts.
Limitations
- Adds a regularization weight hyperparameter lambda that increases computational cost.
- Analysis is limited to CLIP; other VLMs are not studied.
- Outlier selection relies on WordNet and zero-shot CLIP similarity, and near-OOD selection is not clearly better than random.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT WordNet
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: ECE ACE MCE PIECE accuracy harmonic mean
Baselines: CoOp CoCoOp MaPLe DEPT KgCoOp TCP PromptSRC CoPrompt PromptKD ZSCLIP CLIP-adapter VPT
Builds on: CoOp KgCoOp CLIP WordNet temperature scaling
Closest papers by meaning
- Open-Vocabulary Calibration for Fine-tuned CLIP (ICML 2024 · similarity 0.91)
- C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion (ICLR 2024 · similarity 0.91)
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.89)
- Towards Calibrated Robust Fine-Tuning of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models (CVPR 2025 · similarity 0.89)
- NLPrompt: Noise-Label Prompt Learning for Vision-Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.