Research map / Vision-language and multimodal / CLIP and zero-shot
An Empirical Study Into What Matters for Calibrating Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of calibration of 35 vision-language models (mainly zero-shot CLIP variants) showing temperature scaling makes VLMs well calibrated under distribution, label-set, hierarchy, and calibration-set-size shifts, and that synthetic calibration data can work.
Problem
Uncertainty estimation and calibration of vision-language models under distribution shift, label-set changes, label hierarchy changes, and limited calibration data remain under-explored, though VLMs are deployed in risk-sensitive settings.
Approach
Systematically evaluates which factors (calibration distribution, label set, hierarchy level, calibration set size, feature-space distance, prompts, calibration method) affect post-hoc calibration of many VLMs, finding temperature scaling calibrates VLMs robustly and that synthetic images from Stable Diffusion with GPT-3 descriptions can serve as calibration data.
Key results
- Temperature-scaled VLMs reach an average ECE of about 0.05 on OOD test sets while ImageNet-trained models rise to about 0.15 after temperature scaling.
- Calibrating on CIFAR-10-Val or DomainNet-Real markedly lowers ECE on ImageNet-R and ImageNet-S, with linear and rank correlation of calibrated confidence vs accuracy above 0.90.
- On ImageNet-S calibration-by-synthesis with fine-grained synthetic images reduces mean ECE from 0.17 to 0.07 and raises Spearman rho from 0.76 to 0.93.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply temperature scaling | VLMs · ImageNet ID and OOD, including ImageNet-S and ImageNet-R | Average ECE: average ECE, dropping to $0.05$ | Helped |
| Apply temperature scaling | ImageNet-trained models · ImageNet ID and OOD, including ImageNet-S and ImageNet-R | Average ECE: ECE, rising to $0.15$ | Hurt |
| Calibrate on synthetic images generated from fine-grained class labels | VLMs · ImageNet-S | MECE: 0.17 -> 0.07 | Helped |
| Calibrate on synthetic images generated from fine-grained class labels | VLMs · ImageNet-S | MAE: 16.49 -> 6.35 | Helped |
| Calibrate on synthetic images generated from fine-grained class labels | VLMs · ImageNet-S | Spearman's rank correlation (ρ): 0.76 -> 0.93 | Helped |
| Calibrate using one prompt with temperature scaling | MaPLe · ImageNet-S | ECE: 0.0364 -> 0.0073 | Helped |
| Calibrate using one prompt with temperature scaling | MaPLe · ImageNet-V2-A | ECE: 0.0159 -> 0.0165 | Hurt |
| Calibrate using one prompt with temperature scaling | MaPLe · ImageNet-R | ECE: 0.0264 -> 0.0073 | Helped |
| Calibrate using one prompt with temperature scaling | PromptSRC · ImageNet-S | ECE: 0.0323 -> 0.0071 | Helped |
| Calibrate using one prompt with temperature scaling | PromptSRC · ImageNet-V2-A | ECE: 0.0175 -> 0.0173 | Helped |
| Calibrate using one prompt with temperature scaling | PromptSRC · ImageNet-R | ECE: 0.0328 -> 0.0071 | Helped |
| Use a different calibration label set from the target label set | VLMs with temperature scaling · ImageNet-R and ImageNet-S | ECE | Helped |
Surprising findings
- A single randomly chosen prompt is enough to estimate a temperature that calibrates other prompt sets, including machine-generated CuPL prompts.
- Calibration works across mismatched label sets and label hierarchy levels, with calibration error only modestly worse than same-level calibration.
Limitations
- Only classification confidence for vanilla backbone VLMs is studied; detection and segmentation not covered.
- Large language model-based VLMs such as LLaVA are not evaluated.
- Findings on prompt tuning and satellite images are limited to small experiments; ECE shows non-zero residual error under label-set shift.
Experimental setup
Datasets: ImageNet CIFAR-10 DomainNet ImageNet-V2-A ImageNet-R ImageNet-S ObjectNet CIFAR-10.1 CIFAR-10.2 CINIC BREEDS EuroSAT
Benchmarks: ImageNet CIFAR-10 DomainNet ImageNet-V2-A ImageNet-R ImageNet-S ObjectNet CIFAR-10.1 CIFAR-10.2 CINIC EuroSAT
Metrics: expected calibration error R^2 Spearman rank correlation mean absolute error of confidence Frechet inception distance
Baselines: ImageNet-trained models (ResNet, ViT, Swin, ConvNeXt, DeiT) no calibration histogram binning vector scaling spline calibration
Builds on: CLIP temperature scaling CuPL BREEDS
Closest papers by meaning
- Open-Vocabulary Calibration for Fine-tuned CLIP (ICML 2024 · similarity 0.91)
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.90)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.90)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.89)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.89)
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.