Research map / Vision-language and multimodal / CLIP and zero-shot
On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTA, a training-free robust MeanShift test-time augmentation method for CLIP that jointly optimizes density modes and per-view inlierness scores, outperforming test-time prompt tuning (TPT) on 15 datasets with lower runtime.
Problem
Test-time zero-shot generalization of vision-language models like CLIP currently relies on expensive test-time prompt tuning over augmented views, which requires gradients and is impractical for black-box APIs; naive averaging of augmented views is also sensitive to outlier views.
Approach
A robust multi-modal MeanShift over augmented-view embeddings that jointly seeks a density mode and assigns each view an inlierness score (with text-driven affinities), replacing confidence thresholds and prompt optimization with a closed-form block coordinate descent on final embeddings only.
Key results
- MTA averages 63.16% on ImageNet and variants with ViT-B/16 versus 62.40% for TPT, and 64.06% with ensemble.
- MTA averages 64.63% over 10 fine-grained datasets vs. 64.76% for TPT without ensemble, and 65.58% with ensemble vs. 65.16% for CLIP+ensemble.
- MTA is nearly three times faster than TPT per image on ImageNet and improves CoOp (16 shots ImageNet) from 71.87% to 74.20% in few-shot setting with 16 tokens and ProGrad from 72.14% to 74.41%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MTA instead of TPT with RandomCrop views | CLIP ViT-B/16 zero-shot classification · ImageNet variants average | top-1 accuracy: 62.40 -> 63.16 | Helped |
| apply MTA instead of TPT with RandomCrop views | CLIP ViT-B/16 zero-shot classification · ImageNet-R | top-1 accuracy: 77.04 -> 76.92 | Hurt |
| apply MTA instead of TPT | CLIP ViT-B/16 zero-shot classification · 10 fine-grained datasets average | top-1 accuracy: 64.76 -> 64.63 | No clear effect |
| add MTA to CoOp prompts | few-shot CLIP classification · ImageNet, 16 shots | top-1 accuracy: 71.87 -> 74.20 | Helped |
| use diffusion-generated views alongside RandomCrop views | MTA · ImageNet variants average | top-1 accuracy: 62.7 -> 63.28 | Helped |
| add inlierness scores to MeanShift (ablation) | MTA · 15-dataset average | top-1 accuracy: 59.93 -> 64.14 | Helped |
| replace inlierness scores with a 10% confidence threshold (ablation) | MTA · 15-dataset average | top-1 accuracy: 64.14 -> 63.27 | Hurt |
| replace text-based affinity with vision-based affinity (ablation) | MTA · 15-dataset average | top-1 accuracy: 64.14 -> 63.15 | Hurt |
| increase the number of augmented views (ablation) | MTA · ImageNet variants average | top-1 accuracy: +2.3% for 16 views; +4.4% for 128 views | Helped |
| increase the number of augmented views with ensemble prediction (ablation) | MTA + Ensemble · ImageNet variants average | top-1 accuracy: 1.9% gain for batches of 16; 3.1% for larger batches | Helped |
Surprising findings
- TPT fails to outperform the majority-vote ensemble of handcrafted prompts on average across the 10 fine-grained datasets.
- Vision-based affinity measure degrades performance relative to text-based affinity (63.15 vs 64.14 average over 15 datasets).
Limitations
- Gains on some datasets are inconsistent (e.g., EuroSAT drops relative to CLIP), and confidence thresholding peaks on ImageNet-A.
- Relies on text-based affinities from zero-shot predictions, so performance depends on the quality of the text prompts; vision-only affinity is weaker.
- Diffusion-based augmentation is computationally expensive and required changing the rho hyperparameter for that experiment.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch SUN397 Aircraft EuroSAT StanfordCars Food101 OxfordPets Flower102
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch SUN397 Aircraft EuroSAT StanfordCars Food101 OxfordPets Flower102
Metrics: top-1 accuracy runtime per image
Baselines: CLIP CLIP + Ensemble TPT DiffTPT CoOp TPT + CoOp Tip-Adapter Tip-Adapter-F ProGrad
Builds on: TPT MeanShift CoOp DiffTPT Tip-Adapter
Closest papers by meaning
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.92)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.91)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.91)
- Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.