Research map / Label-efficient and robust learning / Test-time adaptation
Efficient Test-Time Adaptation of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TDA, a training-free test-time adaptation method for CLIP that uses dynamic positive and negative key-value caches of test features and pseudo labels, requiring no backpropagation.
Problem
Test-time adaptation of vision-language models like CLIP to distribution shifts is computationally expensive, as prior prompt-tuning methods (TPT, DiffTPT) require backpropagation per test sample, and pseudo labels on test data are noisy.
Approach
A training-free dynamic key-value cache stores test features with entropy-prioritized pseudo labels (positive cache) and adds a negative cache that assigns pseudo labels for classes the sample is unlikely to belong to, when the model is uncertain; both caches are combined with CLIP zero-shot logits.
Key results
- On the OOD benchmark with CLIP-ViT-B/16, TDA reaches 65.01% average accuracy versus 62.44% for TPT.
- On the cross-domain benchmark with CLIP-ViT-B/16, TDA reaches 67.53% average accuracy versus 65.10% for TPT and 65.47% for DiffTPT.
- On ImageNet with CLIP-ResNet-50, TDA runs in 16 minutes versus 12h 50min for TPT and 34h 45min for DiffTPT, with accuracy 61.35% vs 60.74% and 60.80%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace per-sample prompt tuning with a dynamic positive-and-negative cache | CLIP-ResNet-50 · ImageNet | testing time: 12h 50min -> 16min | Helped |
| replace per-sample prompt tuning with a dynamic positive-and-negative cache | CLIP-ResNet-50 · ImageNet | accuracy: 60.74 -> 61.35 | Helped |
| add TDA test-time adaptation | CLIP-ResNet-50 · OOD benchmark (OOD Average) | accuracy: 45.69 -> 46.63 | Helped |
| add TDA test-time adaptation | CLIP-ViT-B/16 · OOD benchmark (OOD Average) | accuracy: 60.81 -> 63.89 | Helped |
| add TDA test-time adaptation | CLIP-ResNet-50 · Cross-domain benchmark (Average) | accuracy: 59.85 -> 61.03 | Helped |
| add TDA test-time adaptation | CLIP-ViT-B/16 · Cross-domain benchmark (Average) | accuracy: 65.47 -> 67.53 | Helped |
| add the positive cache (ablation) | CLIP-ResNet-50 · ImageNet | accuracy: 59.81 -> 60.82 | Helped |
| add the negative cache (ablation) | CLIP-ResNet-50 · ImageNet | accuracy: 59.81 -> 60.83 | Helped |
| combine the positive and negative caches (ablation) | CLIP-ResNet-50 · ImageNet | accuracy: 60.82 -> 61.35 | Helped |
| combine the positive and negative caches (ablation) | TDA · Cross-domain benchmark (Average) | accuracy: 60.38% -> 61.03% | Helped |
| set positive-cache shot capacity to 3 (ablation) | TDA · ImageNet | accuracy | Helped |
| set negative-cache shot capacity to 2 (ablation) | TDA · ImageNet | accuracy | Helped |
Surprising findings
- Larger negative cache shot capacity lowers performance because it stores more high-entropy, noisy pseudo labels.
Limitations
- Pseudo labels are noisy, which the negative cache only partly mitigates.
- Tip-Adapter-style cache cannot handle new classes at test time, so it is excluded from the cross-domain benchmark.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-S Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-S Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Metrics: top-1 accuracy testing time
Baselines: CLIP CoOp CoCoOp Tip-Adapter TPT DiffTPT
Builds on: Tip-Adapter TPT CLIP
Closest papers by meaning
- DOTA: Distributional Test-time Adaptation of Vision-Language Models (NeurIPS 2025 · similarity 0.93)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.93)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.93)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.92)
- BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.