Research map / Vision-language and multimodal / CLIP and zero-shot

O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Test-time adaptation of VLMs · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes O-TPT, which adds an orthogonality regularizer on CLIP text features to test-time prompt tuning (TPT) to reduce calibration error while keeping accuracy.

Problem

Test-time prompt tuning of vision-language models boosts accuracy but leaves predictions poorly calibrated (overconfident), and prior calibration fixes (C-TPT's dispersion loss) fail to sufficiently spread text features.

Approach

Observes that lower pairwise cosine similarity among class text features correlates with lower ECE, and adds a penalty ||EE^T - I||^2 to the TPT loss so text features become near-orthogonal, optionally with a Householder transform on top.

Key results

• With CLIP ViT-B/16, O-TPT reaches average ECE 4.21 across 11 fine-grained datasets versus 5.13 for C-TPT and 4.66 for Robust-adapt-Penalty-CTPT.
• With CLIP RN50, average ECE is 5.45 versus 6.19 for C-TPT.
• On ImageNet OOD variants with ViT-B/16, average ECE is 4.88 versus 5.82 for C-TPT and 12.0 for TPT.

Measured effects reported in the paper

Surprising findings

• Plain TPT's accuracy-to-calibration tradeoff is severe: TPT average ECE 11.6 on ViT-B/16 versus 4.43 for zero-shot CLIP.
• O-TPT is reported to sometimes lose accuracy relative to baselines (e.g., ImageNet-A and ImageNet-R under RN50 and ViT-B/16).

Limitations

• Gains on some datasets are small or negative (e.g., EuroSAT and DTD ECE worse than C-TPT on ViT-B/16), and accuracy sometimes drops slightly.
• Relies on a hand-tuned regularization weight lambda (18 by default, 2 for OOD datasets).
• Correlation between text-feature cosine similarity and calibration is shown empirically, without a theoretical guarantee.

Experimental setup

Datasets: ImageNet Caltech101 DTD Flowers102 Food101 SUN397 FGVC Aircraft OxfordPets UCF101 EuroSAT StanfordCars ImageNet-A

Benchmarks: ImageNet Caltech101 DTD Flowers102 Food101 SUN397 FGVC Aircraft OxfordPets UCF101 EuroSAT StanfordCars ImageNet-A

Metrics: Expected Calibration Error accuracy Static Calibration Error standard deviation across seeds

Baselines: TPT C-TPT Robust-adapt (ZS/SaLS/Penalty)-CTPT Zero-shot CLIP Temperature scaling

Builds on: TPT C-TPT CoOp MAPLE Temperature Scaling

Closest papers by meaning

• C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion (ICLR 2024 · similarity 0.92)
• Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.91)
• Open-Vocabulary Calibration for Fine-tuned CLIP (ICML 2024 · similarity 0.90)
• Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.89)
• Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.89)