Research map / Vision-language and multimodal / CLIP and zero-shot
C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-TPT, which adds a text-feature-dispersion regularizer (maximizing ATFD) to test-time prompt tuning of CLIP to reduce calibration error without labeled data.
Problem
Test-time prompt tuning (TPT) for CLIP improves accuracy via entropy minimization but yields overconfident predictions and higher calibration error (ECE), and standard calibration methods need labeled data.
Approach
Observes that prompts producing higher dispersion of class-embedded text features are better calibrated; defines Average Text Feature Dispersion (ATFD) and adds -ATFD as an auxiliary loss to TPT's entropy objective during test-time tuning, requiring no labels.
Key results
- On fine-grained classification with CLIP-ViT-B/16 (hard prompt init), average ECE drops from 11.6 (TPT) to 5.13 (TPT+C-TPT), a 56% reduction, with accuracy 65.0 to 64.8.
- With CLIP-RN50 (hard prompt init), average ECE falls from 11.7 to 6.20 while accuracy stays near 57.7 to 57.8.
- On ImageNet distribution shifts with CLIP-ViT-B/16 (hard prompt), ECE drops from 12.0 to 5.82 with average accuracy 59.9 to 59.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add C-TPT to TPT with hard-prompt initialization | CLIP-RN50 · Fine-grained classification (average) | ECE: 11.7 -> 6.20 | Helped |
| Add C-TPT to TPT with hard-prompt initialization | CLIP-ViT-B/16 · Fine-grained classification (average) | ECE: 11.6 -> 5.13 | Helped |
| Add C-TPT to TPT with ensemble initialization | CLIP-RN50 · Fine-grained classification (average) | ECE: 10.7 -> 5.20 | Helped |
| Add C-TPT to TPT with ensemble initialization | CLIP-ViT-B/16 · Fine-grained classification (average) | ECE: 9.91 -> 4.62 | Helped |
| Add C-TPT to TPT with hard-prompt initialization | CLIP-RN50 · Natural distribution shifts (average) | ECE: 16.7 -> 12.1 | Helped |
| Add C-TPT to TPT with ensemble initialization | CLIP-RN50 · Natural distribution shifts (average) | ECE: 15.8 -> 13.6 | Helped |
| Add C-TPT to TPT with hard-prompt initialization | CLIP-ViT-B/16 · Natural distribution shifts (average) | ECE: 12.0 -> 5.82 | Helped |
| Add C-TPT to TPT with ensemble initialization | CLIP-ViT-B/16 · Natural distribution shifts (average) | ECE: 10.9 -> 8.32 | Helped |
| Add C-TPT to TPT | CLIP-RN50 and CLIP-ViT-B/16 · Natural distribution shifts (average) | accuracy: 43.5 -> 42.8; 44.5 -> 44.0; 59.9 -> 59.6; 61.2 -> 60.7 | Hurt |
| Use TPT instead of the hard prompt | CLIP · Fine-grained classification | ECE | Hurt |
| Increase the C-TPT weight lambda (ablation) | CLIP-RN50 · Multiple validation datasets | ECE | Helped |
| Use temperature scaling with TPT output logits (ablation) | TPT · Fine-grained classification and natural distribution shifts | ECE | Helped |
Surprising findings
- Hard prompts with similar accuracy can differ substantially in calibration, and poorly calibrated prompts cluster their text features together.
- TPT, which improves accuracy, consistently increases calibration error (ECE) relative to the hard-prompt initialization.
Limitations
- Accuracy can degrade slightly (typically under 1-2%) in some cases due to the calibration regularizer.
- Lambda is fixed at 50.0 (20.0 for distribution shift) without per-sample tuning, which may not be optimal for every sample.
- Hyperparameter selection is constrained because test-time adaptation lacks validation data.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVC Aircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVC Aircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Metrics: accuracy ECE
Baselines: CLIP hard prompt TPT TPT_temp (temperature scaling) CoOp CoCoOp
Builds on: Test-time Prompt Tuning (TPT) CoOp CoCoOp temperature scaling
Closest papers by meaning
- O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models (CVPR 2025 · similarity 0.92)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.91)
- Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.91)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Robust Test-Time Adaptation for Zero-Shot Prompt Tuning (AAAI 2024 · similarity 0.89)
- DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.