Research map / Label-efficient and robust learning / Test-time adaptation
CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPTTA, a test-time adaptation method for CLIP that replaces entropy minimization with a soft contrastive image-text loss (plus regularization, confident memory, and an outlier contrastive exposure loss for open-set), evaluated across 75 datasets.
Problem
Entropy-minimization test-time adaptation is misaligned with CLIP's contrastive image-text pretraining, causing pseudo-label drift and class collapse under distribution shifts.
Approach
Use a soft contrastive loss over batch pseudo-captions that mirrors CLIP's pretraining objective, whose batch-aware softmax lets samples be corrected by other predictions and whose gradient vanishes under collapse; add a negative-marginal-entropy regularizer, a class-wise confident memory, and an OCE loss for ID/OOD separation in open-set settings.
Key results
- On corruptions average, CLIPTTA reaches 58.1% vs 49.8% for CLIPArTT and 42.3% for ETA (Table 1).
- Open-set ImageNet/Places: CLIPTTA+ achieves 97.7 AUC and 9.7 FPR95 versus 95.4 AUC and 17.1 FPR95 for UniEnt.
- Ablation: soft contrastive loss alone improves average over TENT by +11.4 points on the four closed-set benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace entropy-based adaptation with soft contrastive adaptation | CLIP · CIFAR-10-C | accuracy: +20.5 pts | Helped |
| replace entropy-based adaptation with soft contrastive adaptation | CLIP · CIFAR-100-C | accuracy: +17.4 pts | Helped |
| replace entropy-based adaptation with soft contrastive adaptation | CLIP · Imagenet-C | accuracy: +15.6 pts | Helped |
| use CLIPTTA instead of TPT | CLIP ViT-B/16 · corruptions and domain shifts | average accuracy: +19.4 pts | Helped |
| use CLIPTTA instead of TDA | CLIP ViT-B/16 · corruptions and domain shifts | average accuracy: +15.6 pts | Helped |
| add Outlier Contrastive Exposure loss | CLIPTTA · Imagenet with Places OOD | AUC: +2.3 points over UniEnt | Helped |
| add Outlier Contrastive Exposure loss | CLIPTTA · Imagenet with Places OOD | AUC: +8.4 points over SoTTA | Helped |
| add the regularization loss (ablation) | soft contrastive loss · closed-set benchmarks | accuracy: +0.3 pts | Helped |
| add confident memory (ablation) | soft contrastive loss with regularization · closed-set benchmarks | accuracy: +0.8 pts | Helped |
| use soft contrastive loss instead of TENT (ablation) | CLIP · CIFAR-100-C | accuracy: +19.4 points | Helped |
| use soft contrastive loss instead of TENT (ablation) | CLIP · Imagenet-C | accuracy: +22.7 points | Helped |
Surprising findings
- TENT degrades CLIP's zero-shot accuracy on low-accuracy corruption benchmarks (e.g., 17.6% on ImageNet-C vs 25.5% zero-shot)
- TDA lags far behind on corruption datasets despite strong ImageNet-variant results
Limitations
- Only the image-to-text term is used in main experiments; text encoder adaptation is not used by default
- Gradient analysis is for the closed-set loss and binary case; theory is partial
- Baselines required adaptation to CLIP and conditioning for non-episodic setting, which may affect fairness
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CIFAR-10-C CIFAR-100-C ImageNet-C ImageNet VisDA-C PACS OfficeHome ImageNet-Domains SVHN Places365
Benchmarks: CIFAR-10-C CIFAR-100-C ImageNet-C VisDA-C PACS OfficeHome ImageNet-D CIFAR-10 CIFAR-100 ImageNet
Metrics: top-1 accuracy AUC FPR95
Baselines: TENT ETA SAR RoTTA CLIPArTT WATT TPT TDA OSTTA SoTTA STAMP UniEnt
Builds on: TENT CLIP MCM UniEnt CLIPArTT
Closest papers by meaning
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.92)
- What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective (NeurIPS 2026 · similarity 0.91)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.91)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.90)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.