Research map / Label-efficient and robust learning / Test-time adaptation
Evaluation of Test-Time Adaptation Under Computational Time Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a realistic online evaluation protocol for test-time adaptation that feeds methods a constant-speed stream so slower methods skip samples, and benchmarks 12-15 TTA methods showing that simple fast methods often beat slower state-of-the-art ones.
Problem
Current TTA evaluation assumes the stream waits for the adaptation step to finish, ignoring that slow methods miss incoming samples in real deployment, which inflates the value of computationally expensive methods.
Approach
Introduce a relative adaptation speed C(g) = ceil(r / R(g(x))) so that a method that is k times slower than the stream may adapt only on every k-th sample; non-adapting samples are predicted by the unadapted base model. This recovers the offline protocol when C(g)=1.
Key results
- Under the realistic protocol, TTAC-NQ's average error on ImageNet-C rises by 7.5 points and TENT's by 4.3 points, while EATA stays best at 55.6% error.
- BN (2020) achieves 66.7% average error on ImageNet-C under the realistic setup, beating SAR (2023) by 1.7 points.
- RoTTA's error on CIFAR10-C under the realistic protocol rises from 26.5% to 49.0%, worse than the non-adapted source model at 43.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate SHOT with the realistic online protocol instead of the current offline protocol | SHOT · ImageNet-C episodic | error rate: 59.9 -> 59.1; (-0.8) | Helped |
| Evaluate TENT with the realistic online protocol instead of the current offline protocol | TENT · ImageNet-C episodic | error rate: 57.3 -> 61.6; (+4.3) | Hurt |
| Evaluate SAR with the realistic online protocol instead of the current offline protocol | SAR · ImageNet-C episodic | error rate: 56.2 -> 63.4; (+7.2) | Hurt |
| Evaluate EATA with the realistic online protocol instead of the current offline protocol | EATA · ImageNet-C episodic | error rate: 52.0 -> 55.6; (+3.6) | Hurt |
| Evaluate CoTTA with the realistic online protocol instead of the current offline protocol | CoTTA · ImageNet-C episodic | error rate: 61.5 -> 68.0; (+6.5) | Hurt |
| Evaluate TTAC-NQ with the realistic online protocol instead of the current offline protocol | TTAC-NQ · ImageNet-C episodic | error rate: 59.0 -> 66.5; (+7.5) | Hurt |
| Evaluate MEMO with the realistic online protocol instead of the current offline protocol | MEMO · ImageNet-C episodic | error rate: 76.3 -> 81.9; (+5.6) | Hurt |
| Evaluate DDA with the realistic online protocol instead of the current offline protocol | DDA · ImageNet-C episodic | error rate: 64.4 -> 82.0; (+17.6) | Hurt |
| Evaluate SHOT with the realistic online protocol instead of the current offline protocol | SHOT · ImageNet-R | error rate: 70.3 -> 62.6; (+7.7) | Helped |
| Evaluate SHOT with the realistic online protocol instead of the current offline protocol | SHOT · ImageNet-3DCC | error rate: 69.2 -> 67.0; (+2.2) | Helped |
| Evaluate RoTTA with the realistic computationally constrained protocol | RoTTA · CIFAR10-C PTTA | error rate: 26.5 -> 49.0; (-22.5) | Hurt |
| Use realistic online evaluation rather than offline evaluation for continual adaptation | TENT · ImageNet-C continual | error rate: $84.6\pm16.0$ -> $65.6\pm5.0$ | Helped |
Surprising findings
- SHOT improves under the realistic protocol (about 0.8 points lower average error on ImageNet-C) because skipping fine-tuning helps avoid overfitting.
- RoTTA performs worse than the non-adapted source model once computational constraints are enforced.
Limitations
- Relative adaptation speed C(g) depends on hardware timing and is noisy, though the paper averages it and reports stability.
- Single-model evaluation scheme assumes a random classifier for skipped samples, a strict and somewhat artificial choice.
- Results are on image classification benchmarks with ImageNet-style classes, which lack temporal correlation.
Experimental setup
Datasets: ImageNet-C CIFAR10-C ImageNet-R ImageNet-3DCC
Benchmarks: ImageNet-C CIFAR10-C ImageNet-R ImageNet-3DCC
Metrics: error rate average error rate across corruptions continual error rate
Baselines: BN AdaBN SHOT TENT SAR CoTTA LAME EATA TTAC-NQ MEMO DDA
Builds on: TENT SHOT SAR EATA CoTTA
Closest papers by meaning
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.90)
- AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation (CVPR 2024 · similarity 0.89)
- Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment (NeurIPS 2025 · similarity 0.89)
- Unified Entropy Optimization for Open-Set Test-Time Adaptation (CVPR 2024 · similarity 0.88)
- Monitoring Risks in Test-Time Adaptation (NeurIPS 2025 · similarity 0.88)
- Active Test-Time Adaptation: Theoretical Analyses and An Algorithm (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.