Research map / Vision-language and multimodal / CLIP and zero-shot
Bayesian Test-Time Adaptation for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bayesian Class Adaptation (BCA), a backpropagation-free test-time adaptation method for CLIP that updates both class embeddings (likelihood) and per-class priors (posterior-driven) online.
Problem
Test-time adaptation of CLIP for zero-shot image classification under distribution shift, where existing methods adapt only the likelihood (class embeddings) and keep the CLIP prior fixed as a one-hot vector.
Approach
Using Bayes theorem, the posterior is shown to depend on both likelihood and prior; BCA adapts class embeddings with running means (likelihood adaptation) and also updates each class's prior distribution with the incoming posterior (prior adaptation), without backpropagation.
Key results
- On OOD benchmark with ViT-B/16, BCA reaches 64.16 OOD average vs. TDA 63.89 and CLIP 59.42.
- On Cross Domain benchmark with ViT-B/16, BCA averages 68.59 vs. TDA 67.53 and HisTPT 67.6.
- On ImageNet with ResNet-50, BCA runs in 2.42 min at 757M memory vs. TPT's 572.13 min and 21396M, with accuracy 61.81 vs. TDA 61.35.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add likelihood adaptation to CLIP (ablation) | CLIP ResNet-50 · ImageNet | accuracy (%): 59.81 -> 60.95 | Helped |
| Add prior adaptation to CLIP (ablation) | CLIP ResNet-50 · ImageNet | accuracy (%): 59.81 -> 61.15 | Helped |
| Combine likelihood and prior adaptation (ablation) | CLIP ResNet-50 · ImageNet | accuracy (%): 60.95 -> 61.81 | Helped |
| Add likelihood adaptation to CLIP (ablation) | CLIP ViT-B/16 · ImageNet | accuracy (%): 68.34 -> 68.99 | Helped |
| Add prior adaptation to CLIP (ablation) | CLIP ViT-B/16 · ImageNet | accuracy (%): 68.34 -> 69.16 | Helped |
| Combine likelihood and prior adaptation (ablation) | CLIP ViT-B/16 · ImageNet | accuracy (%): 69.16 -> 70.22 | Helped |
| Add likelihood adaptation to CLIP (ablation) | CLIP ResNet-50 · Aircraft | accuracy (%): 16.11 -> 17.85 | Helped |
| Add prior adaptation to CLIP (ablation) | CLIP ResNet-50 · Aircraft | accuracy (%): 16.11 -> 18.10 | Helped |
| Combine likelihood and prior adaptation (ablation) | CLIP ResNet-50 · Aircraft | accuracy (%): 18.10 -> 19.89 | Helped |
| Add likelihood adaptation to CLIP (ablation) | CLIP ViT-B/16 · Aircraft | accuracy (%): 23.22 -> 26.85 | Helped |
| Add prior adaptation to CLIP (ablation) | CLIP ViT-B/16 · Aircraft | accuracy (%): 23.22 -> 26.70 | Helped |
| Combine likelihood and prior adaptation (ablation) | CLIP ViT-B/16 · Aircraft | accuracy (%): 26.85 -> 28.59 | Helped |
Limitations
- Relies on a fixed threshold tau and counters n1, n2 hyperparameters tuned per benchmark.
- Uses the simplified likelihood assumption exp(cos) from original CLIP and uniform class prior P(mu_m) only.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-S Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Metrics: top-1 accuracy testing time memory usage
Baselines: TPT DiffTPT C-TPT TDA MTA PromptAlign HisTPT Zero CLIP
Builds on: CLIP TDA TPT
Closest papers by meaning
- Efficient Test-Time Adaptation of Vision-Language Models (CVPR 2024 · similarity 0.90)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.90)
- Noisy Test-Time Adaptation in Vision-Language Models (ICLR 2025 · similarity 0.90)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.