Research map / Label-efficient and robust learning / Test-time adaptation
Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that test-time adaptation (TTA) strengthens accuracy-on-the-line and agreement-on-the-line trends under distribution shifts where they are otherwise weak, explains this by TTA reducing shifts to feature-space scaling, and uses ALine with TTA for OOD accuracy estimation and hyperparameter/method selection without OOD labels.
Problem
Accuracy-on-the-line and agreement-on-the-line break under some distribution shifts (e.g., CIFAR10-C Gaussian Noise, Camelyon17-WILDS), so ID metrics cannot reliably predict OOD performance and OOD model selection is unreliable.
Approach
Test-time adaptation, although designed to improve OOD accuracy, makes the penultimate-layer feature distribution shift collapse to a scalar scaling of class means and covariances (cosine similarity near 1), which satisfies the Gaussian-linear conditions for accuracy-on-the-line; this restores strong ID-OOD linear trends that ALine can exploit for label-free OOD accuracy estimation and TTA hyperpa
Key results
- Under CIFAR10-C Gaussian Noise, TENT raises the R^2 of ID-OOD accuracy from 0.39 (Vanilla) to 1.0, and ALine estimation MAE drops from 10.76% to 2.34% after adaptation.
- On Camelyon17-WILDS, ALine MAE drops from 12.88% (Vanilla) to 1.42% (ETA with ALine-D).
- Selecting TENT models by best ID accuracy yields 0.62% MAE on Camelyon17-WILDS and 0.36% average MAE on CIFAR10-C, outperforming MixVal, ENT, IM, Corr-C and SND.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply TENT at test time | Vanilla models · Camelyon17-WILDS | agreement R^2: 0.15→0.97 | Helped |
| apply TENT at test time | Vanilla models · Camelyon17-WILDS | accuracy R^2: 0.33→0.97 | Helped |
| apply TENT before ALine-D | OOD accuracy estimation · CIFAR10-C Gaussian Noise | MAE (%): 10.76% → 2.34% | Helped |
| apply ETA before ALine-D | OOD accuracy estimation · Camelyon17-WILDS | MAE (%): 12.88% → 1.42% | Helped |
| apply TENT before ALine-D | OOD accuracy estimation · iWildCAM-WILDS | MAE (%): 2.82 → 2.96 | Hurt |
| apply TTA to models | Vanilla models · CIFAR10-C Gaussian Noise and Camelyon17-WILDS | OOD accuracy estimation MAE (%): 10.76% → 2.34%; 12.88% → 1.42% | Helped |
| select TENT hyperparameters by ID accuracy | TENT · Camelyon17-WILDS | model selection MAE (%): 0.62% | Helped |
| replace BatchNorm with LayerNorm (ablation) | Vanilla models · CIFAR10 vs. CIFAR10-C Gaussian Noise | accuracy R^2: 0.18 → 0.56 | Helped |
| replace BatchNorm with LayerNorm (ablation) | Vanilla models · CIFAR10 vs. CIFAR10-C Gaussian Noise | agreement R^2: 0.39 → 0.86 | Helped |
| remove normalization layers (ablation) | Vanilla models · CIFAR10 vs. CIFAR10-C Gaussian Noise | accuracy R^2: 0.18 → 0.29 | Helped |
| remove normalization layers (ablation) | Vanilla models · CIFAR10 vs. CIFAR10-C Gaussian Noise | agreement R^2: 0.39 → 0.71 | Helped |
| adapt only the final linear classifier instead of modifying the feature extractor (ablation) | T3A · CIFAR10 vs. CIFAR10-C Gaussian Noise | accuracy R^2: 0.99 → 0.80 | Hurt |
Surprising findings
- TTA, though designed to improve OOD accuracy, strengthens linear ACL/AGL trends, even where Vanilla models show very weak correlations.
- Sweeping TTA hyperparameters gives models that lie on the same linear trend, so the best-ID model is a near-optimal OOD choice.
Limitations
- AGL-based estimation still requires sufficient ID labeled data to estimate agreement and accuracy.
- Methods without BN-layer adaptation (e.g., T3A) show weaker linear trends, so the approach may not apply to single-batch settings.
- The theoretical explanation is only partial; AGL is not theoretically justified for the scaled-mean case.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet CIFAR10-C CIFAR100-C ImageNet-C CIFAR10.1 ImageNetV2 ImageNet-R Camelyon17-WILDS iWildCAM-WILDS FMoW-WILDS
Benchmarks: CIFAR10-C CIFAR100-C ImageNet-C CIFAR10.1 ImageNetV2 ImageNet-R Camelyon17-WILDS iWildCAM-WILDS FMoW-WILDS
Metrics: R^2 (correlation coefficient) MAE (%) of OOD accuracy estimate cosine similarity of class means and covariances classification error
Baselines: Vanilla ATC DOC-feat average confidence agreement MixVal ENT IM Corr-C SND T3A
Builds on: accuracy-on-the-line agreement-on-the-line ALine TENT BN_Adapt
Closest papers by meaning
- Evaluation of Test-Time Adaptation Under Computational Time Constraints (ICML 2024 · similarity 0.87)
- AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation (CVPR 2024 · similarity 0.87)
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.87)
- Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment (NeurIPS 2025 · similarity 0.87)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.87)
- Unified Entropy Optimization for Open-Set Test-Time Adaptation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.