Research map / Label-efficient and robust learning / Test-time adaptation
AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AETTA, a label-free method that estimates the accuracy of test-time-adapted models by measuring prediction disagreement with dropout inferences, corrected by a batch-entropy-based robust calibration weight, and uses it for model recovery.
Problem
Estimating the accuracy of test-time adaptation (TTA) models under domain shift without labeled source or target data, since adaptation can fail silently on unlabeled test streams.
Approach
Replaces multiple independently trained models (as in GDE) with dropout inference samples from a single adapted model to estimate error via prediction disagreement, and scales the estimate by a batch-prediction-entropy term that penalizes over-confident, skewed predictions during adaptation failures.
Key results
- AETTA achieves the lowest MAE in most fully and continual TTA settings, averaging 19.8%p more accurate estimation than the four baselines across six TTA methods.
- In the model recovery case study, AETTA-triggered resets achieve an average 11.73%p accuracy improvement, versus 8.74%p for the DistShift oracle and 5.09%p for Episodic.
- Using dropout-inference disagreement without the entropy term (alpha=0) gives notably inaccurate estimates, showing the robust calibration term is needed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AETTA instead of SoftmaxScore for accuracy estimation | TTA methods · Fully TTA, CIFAR10-C | MAE (%): 6.37 ± 0.10 -> 5.05 ± 0.46 | Helped |
| Use AETTA instead of AdvPerturb for accuracy estimation | TTA methods · Fully TTA, CIFAR100-C | MAE (%): 19.91 ± 0.26 -> 8.63 ± 0.24 | Helped |
| Use AETTA instead of AdvPerturb for accuracy estimation | TTA methods · Fully TTA, ImageNet-C | MAE (%): 15.45 ± 0.02 -> 9.55 ± 0.07 | Helped |
| Use AETTA instead of SrcValid for accuracy estimation | TTA methods · Continual TTA, CIFAR10-C | MAE (%): 14.04 ± 0.58 -> 6.00 ± 0.35 | Helped |
| Use AETTA instead of AdvPerturb for accuracy estimation | TTA methods · Continual TTA, CIFAR100-C | MAE (%): 17.16 ± 0.32 -> 5.74 ± 0.13 | Helped |
| Use AETTA instead of AdvPerturb for accuracy estimation | TTA methods · Continual TTA, ImageNet-C | MAE (%): 15.18 ± 0.09 -> 9.24 ± 0.14 | Helped |
| Use AETTA estimates to trigger model resets instead of resetting at known distribution shifts | TTA model recovery · 15 corruption types | Average accuracy improvement (%p): 8.74 ± 0.55 -> 11.73 ± 0.34 | Helped |
| Vary the number of dropout inferences N across 5, 10, and 15 (ablation) | AETTA · Continual CIFAR100-C | MAE (%) | No clear effect |
| Set α = 0, removing robust weighting so Err = PDD (ablation) | AETTA · Continual CIFAR100-C | MAE (%) | Hurt |
| Vary the robust-weighting hyperparameter α (ablation) | AETTA · Continual CIFAR100-C | MAE (%) | Mixed |
Surprising findings
- GDE, which uses adapted models from different stages as the ensemble, performs poorly, suggesting sequential models from one stochastic learning process are not sufficiently i.i.d.
Limitations
- Relies on dropout inference as a proxy for i.i.d. hypotheses, an assumption that may not hold for all architectures.
- Entropy-based weighting uses a hyperparameter alpha (set to 3) that must be tuned; estimates are sensitive to it when set to 0.
- Evaluated only on corruption benchmarks with ResNet18 and image classification.
Experimental setup
Datasets: CIFAR10-C CIFAR100-C ImageNet-C
Benchmarks: CIFAR10-C CIFAR100-C ImageNet-C
Metrics: mean absolute error (MAE) of batch-wise accuracy estimation average accuracy improvement (model recovery)
Baselines: SrcValid SoftmaxScore GDE AdvPerturb Episodic MRS Stochastic FisherStochastic DistShift
Builds on: generalization disagreement equality covariate shift temperature scaling Agreement-on-the-line
Closest papers by meaning
- Evaluation of Test-Time Adaptation Under Computational Time Constraints (ICML 2024 · similarity 0.89)
- Monitoring Risks in Test-Time Adaptation (NeurIPS 2025 · similarity 0.89)
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.89)
- ReservoirTTA: Prolonged Test-time Adaptation for Evolving and Recurring Domains (NeurIPS 2025 · similarity 0.88)
- Buffer layers for Test-Time Adaptation (NeurIPS 2025 · similarity 0.87)
- Bayesian Weight Enhancement with Steady-State Adaptation for Test-time Adaptation in Dynamic Environments (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.