Research map / Continual learning / Continual fine-tuning of LLMs
DOTA: Distributional Test-time Adaptation of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOTA, a cache-free test-time adaptation method for CLIP that streams estimates of per-class Gaussian distributions (EM-style, weighted by zero-shot probabilities) and classifies via Bayes' posterior fused with zero-shot logits.
Problem
Cache-based test-time adapters for vision-language models store a limited set of test samples and suffer catastrophic forgetting when samples are dropped, limiting performance under distribution shift.
Approach
Instead of memorizing individual test samples, continuously estimate per-class Gaussian distribution parameters (mean and covariance) from the test stream, weighting each sample by its zero-shot posterior (single EM iteration), and classify with Bayes' posterior adaptively fused with zero-shot logits.
Key results
- Cross-domain average over 10 datasets with ViT-B/16: 69.71% vs TDA 67.53% and zero-shot 64.59%.
- NDS average 66.31% vs TDA 65.10% with CLIP ViT-B/16 (ImageNet 70.69 vs TDA 69.51).
- On PLIP pathology datasets, average 70.56% (CTA column) vs TDA 63.67% and zero-shot 61.80%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace cache-based sample storage with continual distribution estimation | CLIP ViT-B/16 · Cross-domain generalization average | top-1 accuracy (%): 68.68 -> 69.71 | Helped |
| Replace cache-based sample storage with continual distribution estimation | CLIP ViT-B/16 · Natural distribution shifts average | top-1 accuracy (%): 65.10 -> 66.31 | Helped |
| Apply Dota adaptation | PLIP · PLIP pathology average | accuracy: 63.67 -> 70.56 | Helped |
| Apply Dota instead of zero-shot classification | CLIP ViT-B/16 · ImageNet | accuracy: 68.34 -> 70.69 | Helped |
| Apply Dota instead of TDA | CLIP ViT-B/16 · ImageNetV2 (10 samples per class) | accuracy: 64.67 -> 64.50 | Hurt |
| Estimate distributions using all zero-shot predictive probabilities instead of only high-confidence samples and predictive labels (ablation) | Dota · Cross-domain generalization average | top-1 accuracy (%): 67.22 -> 69.71 | Helped |
| Update the covariance matrix rather than remove covariance updates (ablation) | Dota · Cross-domain generalization average | top-1 accuracy (%): 67.32 -> 69.71 | Helped |
| Vary sigma^2 across the evaluated values (ablation) | Dota · ImageNet | accuracy: 70.42 to 70.72 | No clear effect |
| Vary rho and eta in the reported combinations (ablation) | Dota · ImageNet | accuracy: 70.69 to 70.00 | Hurt |
| Increase the number of observed test samples by evaluating the last 50% rather than all samples | Dota · ImageNetV2 (10 samples per class), ViT-B/16 | accuracy: 64.50 -> 65.20 | Helped |
Surprising findings
- On ImageNetV2 with only 10 samples per class, DOTA slightly underperforms TDA (64.50 vs 64.67 for ViT-B/16).
- Performance on the last 50% of the stream is noticeably higher than on the full stream, while TDA declines on several datasets.
Limitations
- Does not consistently beat TDA on datasets with few samples per class (e.g., ImageNetV2 with 10 samples per class).
- Covariance is approximated by a shared average across classes, and results depend on hyperparameters (sigma^2, rho, eta) tuned on validation sets.
- Gains are reported for CLIP backbones mainly; the cross-domain tables rely on numbers from original papers.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-R ImageNet-S ImageNet-V2 Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Benchmarks: ImageNet ImageNet-A ImageNet-R ImageNet-S ImageNet-V2 Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Metrics: top-1 accuracy inference time
Baselines: Zero-Shot CLIP TPT DiffTPT TDA BoostAdapter HisTPT ZERO DMN
Builds on: TDA CLIP TPT T3A Gaussian discriminant analysis
Closest papers by meaning
- Efficient Test-Time Adaptation of Vision-Language Models (CVPR 2024 · similarity 0.93)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.93)
- BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping (NeurIPS 2024 · similarity 0.92)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.91)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.91)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.