Research map / Vision-language and multimodal / CLIP and zero-shot
Realistic Test-Time Adaptation of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a realistic evaluation framework for transductive/test-time adaptation of VLMs (variable effective classes per batch, correlated online streams) and StatA, a Gaussian MLE clustering method with a text-derived KL statistical anchor that stays robust across scenarios.
Problem
Existing transductive and test-time adaptation methods for vision-language models assume favorable data distributions (all classes present, uniform classes, i.i.d. streams), and degrade the zero-shot robustness of CLIP when these assumptions fail.
Approach
Regularize the Gaussian class statistics (means and covariances) of the vision features toward text-derived anchors via a KL term (StatA), in addition to the usual Laplacian and text-supervision assignment regularizers, so the method stays robust when the number of effective classes per batch varies.
Key results
- StatA reaches 70.4% average accuracy in the Low-Keff batch-64 setting (vs CLIP 65.2%), while Dirichlet and ZLaP drop well below zero-shot in some regimes.
- In the batch-1000 Very High (50-100 classes) setting, StatA gets 69.0% vs TransCLIP 64.4% and Dirichlet 33.6%.
- StatA processes 50,000 samples in about 15 seconds on an RTX 3090 (ImageNet).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply StatA instead of zero-shot CLIP | CLIP ViT-B/16 · Batch adaptation across Very Low, Low, Medium, High, Very High, and All effective-class scenarios | average accuracy: 70.4+5.1; 69.3+4.1; 67.4+2.2; 69.8+4.5; 69.0+3.7; 69.9+4.7 | Helped |
| apply TransCLIP instead of zero-shot CLIP | CLIP ViT-B/16 · Very Low (1–4 effective classes), batch size 64 | average accuracy: 37.7-29.8 | Hurt |
| apply Dirichlet instead of zero-shot CLIP | CLIP ViT-B/16 · Low (2–10 effective classes), batch size 64 | average accuracy: 70.3+5.1 | Helped |
| apply Dirichlet instead of zero-shot CLIP | CLIP ViT-B/16 · High (25–50 effective classes), batch size 1,000 | average accuracy: 43.6-23.9 | Hurt |
| apply TransCLIP instead of zero-shot CLIP | CLIP ViT-B/16 · All classes, whole dataset | average accuracy: 70.3+5.1 | Helped |
| apply StatA instead of zero-shot CLIP | CLIP ViT-B/16 · Online adaptation, Low, Medium, and High correlation and Separate sequential classes | average accuracy: 67.0+1.7; 68.9+3.7; 69.5+4.2; 69.1+3.8 | Helped |
| increase anchor weighting α, especially when effective class count is low relative to batch size (ablation) | StatA · ImageNet, batch size 64 | accuracy: +10%; +13% | Helped |
| vary anchor weighting α (ablation) | StatA · DTD and Cars, batch size 1,000, as effective class count increases | accuracy | Mixed |
| use hard rather than soft assignments to compute βk (ablation) | StatA · ImageNet, batch size 1,000, effective classes 5, 10, 25, 50, 100 | accuracy: 70.3 -> 71.3; 69.2 -> 70.2; 70.7 -> 71.4; 72.7 -> 72.2; 72.8 -> 71.4 | Mixed |
| use hard rather than soft assignments to compute βk (ablation) | StatA · ImageNet, batch size 5,000, effective classes 50, 100, 250, 500, 1,000 | accuracy: 69.7 -> 70.8; 69.2 -> 70.4; 70.8 -> 71.1; 69.4 -> 69.5; 66.1 -> 66.5 | Mixed |
Surprising findings
- Existing transductive methods such as Dirichlet and ZLaP fall far below zero-shot CLIP (e.g., ZLaP 27.5% vs 65.2% in Very Low setting), showing they can harm zero-shot robustness.
- TENT does not consistently improve CLIP, plausibly because CLIP has no batch normalization layers.
Limitations
- Performance depends on the anchor weight alpha, which the authors tune per scenario in ablations and leave adaptive selection to future work.
- Some scenarios (e.g., Aircraft, Caltech101 under certain settings) show small or no gains over zero-shot.
Experimental setup
Datasets: ImageNet SUN397 FGVC Aircraft EuroSAT Stanford Cars Food101 Oxford Pets Flowers102 Caltech101 DTD UCF101
Benchmarks: ImageNet SUN397 FGVC Aircraft EuroSAT Stanford Cars Food101 Oxford Pets Flowers102 Caltech101 DTD UCF101
Metrics: accuracy averaged over tasks
Baselines: CLIP zero-shot MTA Dirichlet ZLaP TransCLIP TENT TDA DMN
Builds on: TransCLIP PADDLE LaplacianShot Dirichlet (EM-Dirichlet) CoOp
Closest papers by meaning
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.94)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.93)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.92)
- Efficient Test-Time Adaptation of Vision-Language Models (CVPR 2024 · similarity 0.92)
- Noisy Test-Time Adaptation in Vision-Language Models (ICLR 2025 · similarity 0.91)
- DOTA: Distributional Test-time Adaptation of Vision-Language Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.