Research map / Label-efficient and robust learning / Test-time adaptation
Robust Test-Time Adaptation for Zero-Shot Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADAPROMPT, a training-free-style test-time prompt tuning method for CLIP that ensembles multiple hand-crafted prompts, tunes them on unlabeled test streams, and uses a confidence-aware class-balanced buffer to reduce error accumulation.
Problem
Zero-shot CLIP classification depends on hand-crafted prompts that are hard to choose without data, and existing test-time prompt tuning (TPT) degrades on some domains under distribution shift while being costly due to heavy augmentation.
Approach
Identifies "Data Bias" (optimal prompt varies across domains) and "Model Bias" (CLIP's biased predictions cause error accumulation); addresses them by ensembling multiple prompts, tuning all prompts with cross-entropy on pseudo-labels, and filtering pseudo-labeled samples through a confidence-aware class-balanced buffer, avoiding costly test-time augmentation.
Key results
- On CIFAR10-C at corruption level 3, ADAPROMPT reaches 79.15% average accuracy vs 76.29% for TPT and 75.90% for the source CLIP baseline.
- On CIFAR10-C level 5, ADAPROMPT averages 66.12% vs 63.81% for TPT; on TinyImageNet-C level 3 it gains 2.22 points over TPT (31.42 vs 29.20).
- Time cost on CIFAR10-C level 5 is 2143.8 s for ADAPROMPT versus 41257.35 s for TPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ADAPROMPT instead of TPT | CLIP ViT-B/16 · CIFAR10-C, corruption level 3 (average) | accuracy: 76.29 -> 79.15 | Helped |
| Use ADAPROMPT instead of TPT | CLIP ViT-B/16 · CIFAR10-C, corruption level 5 (average) | accuracy: 63.81 -> 66.12 | Helped |
| Use ADAPROMPT instead of TPT | CLIP ViT-B/16 · CIFAR100-C, corruption level 3 (average) | accuracy: 49.54 -> 50.93 | Helped |
| Use ADAPROMPT instead of TPT | CLIP ViT-B/16 · CIFAR100-C, corruption level 5 (average) | accuracy: 36.07 -> 37.44 | Helped |
| Use ADAPROMPT instead of TPT | CLIP ViT-B/16 · TinyImageNet-C, corruption level 3 (average) | accuracy: 29.20 -> 31.42 | Helped |
| Use ADAPROMPT instead of TPT | CLIP ViT-B/16 · ImageNet-R | accuracy: 74.19 -> 73.98 | Hurt |
| Add prompt ensembling to the best single prompt, with updates (ablation) | ADAPROMPT · CIFAR10-C, corruption level 3 (average) | accuracy: 77.72 ± 0.24 -> 79.15 ± 0.23 | Helped |
| Add prompt ensembling to the best single prompt, with updates (ablation) | ADAPROMPT · CIFAR10-C, corruption level 5 (average) | accuracy: 65.32 ± 0.18 -> 66.12 ± 0.43 | Helped |
| Add prompt ensembling without updating prompts (ablation) | ADAPROMPT · CIFAR10-C, corruption level 3 (average) | accuracy: 76.21 ± 0.00 -> 75.38 ± 0.00 | Hurt |
| Add prompt ensembling without updating prompts (ablation) | ADAPROMPT · CIFAR10-C, corruption level 5 (average) | accuracy: 62.77 ± 0.00 -> 61.75 ± 0.00 | Hurt |
| Add buffer-based prompt updates without ensembling (ablation) | ADAPROMPT · CIFAR10-C, corruption level 3 (average) | accuracy: 76.21 ± 0.00 -> 77.72 ± 0.24 | Helped |
| Add buffer-based prompt updates without ensembling (ablation) | ADAPROMPT · CIFAR10-C, corruption level 5 (average) | accuracy: 62.37 ± 0.00 -> 65.32 ± 0.18 | Helped |
Surprising findings
- TPT-Continual collapses on some domains (e.g., about 0.5% accuracy on CIFAR10-C), showing that continual prompt tuning suffers strongly from model bias.
- On ImageNet-R, ADAPROMPT (73.98%) does not outperform TPT (74.19%).
Limitations
- Degrades to ImageNet-R performance slightly below TPT.
- Evaluated mainly on corrupted CIFAR-style benchmarks with a small prompt set of three hand-crafted prompts.
Experimental setup
Datasets: CIFAR10-C CIFAR100-C TinyImageNet-C ImageNet-R
Benchmarks: CIFAR10-C CIFAR100-C TinyImageNet-C ImageNet-R
Metrics: accuracy time cost (s)
Baselines: CLIP zero-shot TPT TPT-Continual
Builds on: TPT CLIP CoOp EATA
Closest papers by meaning
- DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025 · similarity 0.90)
- C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion (ICLR 2024 · similarity 0.89)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.88)
- Understanding prompt engineering may not require rethinking generalization (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.