Research map / Vision-language and multimodal / CLIP and zero-shot
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, Chaowei Xiao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune the prompt on a single test image using confidence-selected augmented views | CLIP-RN50 · ImageNet variants (OOD Average) | Top-1 accuracy: 40.69 -> 43.89 | Improved |
| Tune the prompt on a single test image using confidence-selected augmented views | CLIP-ViT-B/16 · ImageNet variants (OOD Average) | Top-1 accuracy: 57.2 -> 60.81 | Improved |
| Initialize test-time tuning from a CoOp prompt | CLIP-RN50 · ImageNet variants (OOD Average) | Top-1 accuracy: 42.43 -> 45.75 | Improved |
| Tune prompts per test sample rather than use an ImageNet-trained prompt | CLIP-RN50 · ImageNet to fine-grained datasets (Average) | Top-1 accuracy: 57.23 -> 57.66 | Improved |
| Tune prompt and class tokens on the support images of each reasoning task | CLIP-RN50 · Bongard-HOI (Average) | accuracy: 62.46 -> 66.59 | Improved |
| Add confidence selection to entropy-minimizing TPT (ablation) | CLIP-RN50 · ImageNet variants (OOD Average) | Top-1 accuracy: 42.28 -> 43.89 (+1.61) | Improved |
| Use top-10% confident views rather than a less selective cutoff (ablation) | TPT · ImageNet variants | Top-1 accuracy | Improved |
| Optimize the text prompt instead of other CLIP parameter groups (ablation) | CLIP test-time optimization · ImageNet variants | Top-1 accuracy | Improved |
| Optimize the visual encoder instead of the text prompt (ablation) | CLIP test-time optimization · ImageNet variants | Top-1 accuracy | Hurt |
| Increase augmented views from 8 to 64 (ablation) | TPT · Natural distribution shifts | Average accuracy: over 2% | Improved |
| Increase optimization steps from 1 to 2 (ablation) | TPT · Natural distribution shifts | Average accuracy: 0.4% | Improved |
| Increase optimization beyond 2 steps (ablation) | TPT · Natural distribution shifts | Average accuracy | No clear effect |
Closest papers by meaning
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.94)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.93)
- DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025 · similarity 0.91)
- Robust Test-Time Adaptation for Zero-Shot Prompt Tuning (AAAI 2024 · similarity 0.91)
- Any-Shift Prompting for Generalization over Distributions (CVPR 2024 · similarity 0.91)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
