Research map / Vision-language and multimodal / CLIP and zero-shot
Open-Vocabulary Calibration for Fine-tuned CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that prompt-tuned CLIP is overconfident on novel classes where post-hoc calibration fails, and proposes Distance-Aware Calibration (DAC), a training-free temperature rescaling by a text-embedding deviation score relative to base classes.
Problem
Confidence calibration of fine-tuned (prompt-tuned) vision-language models degrades in the open-vocabulary setting: post-hoc calibration fitted on base classes does not transfer to novel classes.
Approach
Scale the CLIP logit temperature per predicted class by a textual deviation (TD) score, the ratio of K-nearest-neighbor proximity of the tuned text embedding to base-class text embeddings versus the pre-trained CLIP's; low-proximity (novel, far from base) classes get higher temperature. Uses only text labels, no visual data from novel classes, and keeps base-class behavior identical to temperature
Key results
- On 11 datasets, DAC reduces average ECE for CoOp from 13.84% to 7.00% on novel classes, with up to 16% reduction on individual datasets.
- On DTD, DAC reduces CoOp's gap in the 0.9-confidence bin from 41.60 to 11.37.
- DAC is applied to 7 prompt/multi-modal tuning methods and consistently lowers novel-class ECE, and it also improves Density-Ratio Calibration (ImageNet ECE_new 8.08 to 3.29 combined).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DAC to CoOp | CoOp · 11 datasets, new classes (average) | ECE: 13.84 -> 7.00 | Helped |
| Apply DAC to CoCoOp | CoCoOp · 11 datasets, new classes (average) | ECE: 6.29 -> 4.82 | Helped |
| Apply DAC to ProDA | ProDA · 11 datasets, new classes (average) | ECE: 4.27 -> 3.99 | Helped |
| Apply DAC to KgCoOp | KgCoOp · 11 datasets, new classes (average) | ECE: 4.36 -> 4.32 | Helped |
| Apply DAC to MaPLe | MaPLe · 11 datasets, new classes (average) | ECE: 5.77 -> 4.61 | Helped |
| Apply DAC to ProGrad | ProGrad · 11 datasets, new classes (average) | ECE: 4.22 -> 3.74 | Helped |
| Apply DAC to PromptSRC | PromptSRC · 11 datasets, new classes (average) | ECE: 3.84 -> 3.63 | Helped |
| Combine DAC with DEN | CoOp + DEN · ImageNet-1K, new classes | ECE: 8.08 -> 3.29 | Helped |
| Apply DAC to FLYP | FLYP · 11 datasets, new classes (average) | ECE: 13.27 -> 10.40 | Helped |
| Use K greater than 0 textual neighbors for proximity estimation (ablation) | DAC · DTD and SUN397 | ECE | Helped |
| Normalize textual features when calculating TD scores (ablation) | DAC with CoOp · Open-vocabulary calibration | ECE | Helped |
| Calibrate using base-class data rather than novel-class data (ablation) | Temperature Scaling on CoOp · 11 datasets, base classes | ECE: 2.95 -> 2.57 | Helped |
Surprising findings
- Tuned VLMs are underconfident on base classes but overconfident on novel classes.
- Standard temperature scaling calibrated on base classes worsens novel-class calibration (e.g., CoOp ECE_new rises to 13.05 on new classes for TS_base).
Limitations
- Evaluated only on CLIP-based prompt/multi-modal tuning and one full fine-tuning method (FLYP), mostly with ViT-B/16 and limited backbones.
- Relies on a choice of K nearest neighbors and a calibration set drawn from base classes for baseline methods.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Metrics: ECE ACE MCE PIECE
Baselines: Temperature Scaling Density-Ratio Calibration Histogram Binning Isotonic Regression Multi-Isotonic Regression zero-shot CLIP
Builds on: CLIP CoOp CoCoOp Temperature Scaling Density-Ratio Calibration
Closest papers by meaning
- Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.91)
- An Empirical Study Into What Matters for Calibrating Vision-Language Models (ICML 2024 · similarity 0.91)
- O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models (CVPR 2025 · similarity 0.90)
- C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion (ICLR 2024 · similarity 0.89)
- PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models (CVPR 2025 · similarity 0.88)
- Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.