Research map / Vision-language and multimodal / CLIP and zero-shot

Frustratingly Easy Test-Time Adaptation of Vision-Language Models

NeurIPS 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Test-time adaptation of VLMs · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that test-time prompt tuning via marginal entropy minimization barely changes the marginal prediction, and proposes ZERO, a training-free TTA baseline that sets softmax temperature to zero, confidence-filters augmented views, and majority-votes them.

Problem

Episodic test-time adaptation of vision-language models usually relies on marginal entropy minimization with backpropagation, which slows inference and increases memory, and its actual benefit over the augmentation-marginalized prediction is unclear.

Approach

Theoretically and empirically show MEM leaves the argmax of the marginal distribution largely unchanged; augmentation-induced overconfidence hurts the marginal's reliability; so set softmax temperature to zero on confidence-filtered augmented views and marginalize (equivalent to majority voting), requiring only a forward pass and no backprop.

Key results

• ZERO reaches 63.74 mean top-1 on natural distribution shifts with CLIP-ViT-B-16 vs 62.44 for TPT; ZERO+Ensemble reaches 66.10.
• ZERO is about 9.5x faster than TPT and uses 12.61x less peak memory on ImageNet with CLIP-ViT-B-16 (0.06 s vs 0.57 s per image; 1.40 GB vs 17.66 GB).
• With CLIP-ViT-B-16 alone, ZERO underperforms TPT on fine-grained classification (64.21 vs 64.78 mean), while ZERO+Ensemble reaches 65.52, outperforming TPT.

Measured effects reported in the paper

Surprising findings

• MEM changes the argmax of the marginal distribution in only a few percent of samples (invariance ratio 91-97% across ImageNet variants), so it has little effect on the final prediction.
• Zero-temperature marginalization performs worse than zero-shot CLIP on EuroSAT, identified as a consistent failure case.

Limitations

• Relies on empirical observations (augmentation-induced overconfidence, comparable error rates) that may not persist for other VLMs or benchmarks; fails on EuroSAT.
• Theoretical invariance proposition relies on assumptions (global entropy minimization, first-order Taylor approximations) that may not hold in general.
• Compute scales linearly with the number of augmented views since each view is forwarded through the vision encoder.

Experimental setup

Datasets: ImageNet-1k ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Oxford Flowers DTD Oxford-IIIT Pets Stanford Cars UCF101 Caltech101 Food101

Benchmarks: ImageNet-1k ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Oxford Flowers DTD Oxford-IIIT Pets Stanford Cars UCF101 Caltech101 Food101

Metrics: top-1 accuracy runtime per image peak GPU memory expected calibration error

Baselines: TPT PromptAlign RLCF Zero-Shot CLIP Ensemble

Builds on: Test-Time Prompt Tuning MEMO CoOp MaPLe CLIP

Closest papers by meaning

• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.94)
• Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models (NeurIPS 2025 · similarity 0.93)
• Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.93)
• On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.92)
• Efficient Test-Time Adaptation of Vision-Language Models (CVPR 2024 · similarity 0.92)
• Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.92)