Research map / Label-efficient and robust learning / Test-time adaptation
Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADAPT, a backpropagation-free, source-free test-time adaptation method for CLIP that models class-conditional features as shared-covariance Gaussians, with closed-form updates and knowledge-bank regularization, supporting both online and transductive settings.
Problem
Test-time adaptation of vision-language models like CLIP under distribution shift is often gradient-based or iterative, and existing training-free methods lack explicit class-conditional distribution modeling, limiting scalability and robustness in both online and transductive settings.
Approach
Reframes TTA as Gaussian probabilistic inference (GDA with shared covariance), progressively estimating class means from CLIP prototypes and per-class high-confidence knowledge banks, and deriving a closed-form, one-pass label estimator regularized by CLIP priors and bank-based consistency.
Key results
- Online setting: ADAPT reaches 66.53% average accuracy on natural distribution shift benchmarks (OOD avg 65.44%), the best among online methods reported.
- Transductive setting: ADAPT reaches 67.09% average on natural distribution shifts, above Frolic (66.00%) and TransCLIP (61.36%).
- On ImageNet, ADAPT online takes 1h 11m and 0.93GB versus TPT at 9h 45m and 4.29GB; transductive variant takes 0.73m and 3.37GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online ADAPT rather than the strongest listed online baseline | CLIP ViT-B/16 · Natural distribution shift, online OOD average | Top-1 accuracy (%): 64.53 -> 65.44 | Helped |
| Use transductive ADAPT rather than Frolic | CLIP ViT-B/16 · Natural distribution shift, transductive average | Top-1 accuracy (%): 66.00 -> 67.09 | Helped |
| Use ADAPT rather than the strongest listed online corruption baseline | CLIP ViT-B/16 · ImageNet-C, online average | Top-1 accuracy (%): 28.34 -> 28.56 | Helped |
| Use transductive ADAPT rather than TransCLIP | CLIP ViT-B/16 · ImageNet-C, transductive average | Top-1 accuracy (%): 27.07 -> 30.29 | Helped |
| Use online ADAPT rather than DMN | CLIP ViT-B/16 · Fine-grained categorization, online average | Top-1 accuracy (%): 70.30 -> 70.76 | Helped |
| Use transductive ADAPT rather than Frolic | CLIP ViT-B/16 · Fine-grained categorization, transductive average | Top-1 accuracy (%): 71.10 -> 72.43 | Helped |
| Replace iterative optimization with closed-form inference (ablation) | ADAPT, online · Task 1 natural distribution shift; Task 2 corruption robustness; Task 3 fine-grained categorization | Accuracy: 66.74 -> 66.53; 28.68 -> 28.56; 71.05 -> 70.76 | Mixed |
| Replace iterative optimization with closed-form inference (ablation) | ADAPT, transductive · Task 1 natural distribution shift; Task 2 corruption robustness; Task 3 fine-grained categorization | Accuracy: 66.88 -> 67.09; 27.95 -> 30.29; 73.98 -> 72.43 | Mixed |
| Initialize class means with GPT-generated descriptions rather than Vanilla prototypes (ablation) | ADAPT · Task 1 natural distribution shift | Accuracy: 64.70 -> 66.53 | Helped |
| Use GPT & Ensemble instead of GPT mean initialization (ablation) | ADAPT · Task 2 corruption robustness | Accuracy: 28.56 -> 28.98 | Helped |
| Use easy-to-hard rather than random input ordering (ablation) | ADAPT, online · Task 1 natural distribution shift | Accuracy: 66.53 -> 66.79 | Helped |
| Use hard-to-easy rather than random input ordering (ablation) | ADAPT, online · Task 1 natural distribution shift | Accuracy: 66.53 -> 65.37 | Hurt |
Surprising findings
- Closed-form one-pass updates match or slightly exceed iterative optimization in transductive accuracy (e.g., 30.29 vs 27.95 on corruption robustness), though online closed-form is slightly lower than iterative on Task 1 and 3.
- Class-specific covariance matrices reportedly degrade accuracy significantly versus the shared covariance due to data sparsity.
Limitations
- Gaussian shared-covariance assumption may fail to capture multi-modal or complex class distributions.
- Method relies on hyperparameters such as knowledge bank size L and alpha, and online results are sensitive to small L.
- Online performance is below iterative optimization on some tasks, and ADAPT trails the Oracle by about 5% on fine-grained tasks.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-R ImageNet-V ImageNet-Sketch ImageNet-C Aircraft Caltech101 Cars DTD EuroSAT Flower102
Benchmarks: ImageNet ImageNet-A ImageNet-R ImageNet-V ImageNet-Sketch ImageNet-C Aircraft Caltech101 Cars DTD EuroSAT Flower102
Metrics: top-1 accuracy time memory
Baselines: CLIP Tip-Adapter TPT DiffTPT C-TPT DMN DPE TPS DynaPrompt B2TPT MTA TDA
Builds on: Gaussian discriminant analysis Tip-Adapter TDA TransCLIP TPT
Closest papers by meaning
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.92)
- Bayesian Weight Enhancement with Steady-State Adaptation for Test-time Adaptation in Dynamic Environments (ICML 2025 · similarity 0.90)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.89)
- Backpropagation-free Network for 3D Test-time Adaptation (CVPR 2024 · similarity 0.89)
- Evaluation of Test-Time Adaptation Under Computational Time Constraints (ICML 2024 · similarity 0.89)
- EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.