Research map / Label-efficient and robust learning / Test-time adaptation
Noisy Test-Time Adaptation in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Zero-Shot Noisy TTA (ZS-NTTA) for CLIP and proposes AdaND, which freezes the classifier and trains a lightweight noise detector on ZS-CLIP pseudo-labels with injected Gaussian noise, improving ZS-NTTA and zero-shot OOD detection.
Problem
Test-time adaptation of vision-language models on data streams containing noisy samples (outside the ID label space) degrades performance, often below the frozen zero-shot model; the task requires online clean/noisy detection plus ID classification.
Approach
Decouple the classifier from the noise detector: keep CLIP frozen as the classifier and train only a single linear noise detector online, using frozen ZS-CLIP detections as pseudo-labels, and inject Gaussian noise samples so the detector does not misflag clean streams as noisy.
Key results
- On ImageNet, AdaND improves average Acc_H by 8.32% over compared TTA methods in ZS-NTTA (73.09 vs 64.77 for ZS-CLIP).
- For ZS-OOD detection on ImageNet, AdaND reduces average FPR95 by 9.40% relative to prior methods, reaching 16.00 vs 25.40 for NegLabel, with AUROC 95.58.
- Existing TTA methods such as Tent (Normal pipeline) drop well below ZS-CLIP on CIFAR-10 (average Acc_H 77.78 vs 86.47), while AdaND reaches 93.34.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaND instead of ZS-CLIP for noisy test-time adaptation | CLIP ViT-B/16 on ImageNet · ImageNet / iNaturalist, SUN, Texture, Places (Avg) | Acc_H: 64.77 -> 73.09 | Helped |
| Use AdaND instead of the strongest reported OOD baseline | CLIP ViT-B/16 on ImageNet · ImageNet / iNaturalist, SUN, Texture, Places (Avg) | FPR95: 25.40 -> 16.00 | Helped |
| Add Gaussian noise injection to AdaND (ablation) | AdaND on ImageNet · Clean data stream | Acc_S: 37.54 -> 63.96 | Helped |
| Add Gaussian noise injection to AdaND (ablation) | AdaND on ImageNet · Noisy data stream | Acc_H: 73.00 -> 73.09 | No clear effect |
| Add a noise detector to the Gaussian-noise-only variant (ablation) | AdaND on ImageNet · Noisy data stream | Acc_H: 64.95 -> 73.09 | Helped |
| Add a noise detector and Gaussian noise injection (ablation) | Frozen CLIP on ImageNet · Noisy data stream | Acc_H: 64.77 -> 73.09 | Helped |
| Remove injected Gaussian noise (ablation) | AdaND on ImageNet with NINCO · Noisy data stream | Acc_H: 57.82 -> 61.12 | Mixed |
Surprising findings
- Most TTA methods (Tent, TPT) underperform the frozen zero-shot CLIP once noisy samples are present, even when filtered with an OOD score and adaptive threshold.
- Noisy samples produce gradients often an order of magnitude larger than clean samples, causing overfitting to noise during adaptation.
Limitations
- Relies on zero-shot CLIP detection accuracy for pseudo-labels; fails when ZS-CLIP OOD detection is poor.
- Using the noise detector's own outputs as pseudo-labels can cause large drops on some ID datasets due to cumulative errors.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CUB-200-2011 Stanford Cars Food-101 Oxford-IIIT Pet ImageNet ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch SVHN
Benchmarks: ZS-NTTA benchmark (CIFAR-10/100, CUB, Stanford Cars, Food-101, Oxford Pet, ImageNet variants with OOD sets) ZS-OOD detection on ImageNet (iNaturalist, SUN, Texture, Places) NINCO ImageNet-C CIFAR-10-C CIFAR-100-C
Metrics: Acc_S Acc_N Acc_H AUROC FPR95
Baselines: ZS-CLIP Tent SoTTA TPT Energy MaxLogit MCM CLIPN NegLabel TDA
Builds on: MCM OWTTT TPT Tent SoTTA
Closest papers by meaning
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.91)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.91)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Efficient Test-Time Adaptation of Vision-Language Models (CVPR 2024 · similarity 0.90)
- Bayesian Test-Time Adaptation for Vision-Language Models (CVPR 2025 · similarity 0.90)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.