Research map / Label-efficient and robust learning / Test-time adaptation
Test-Time Model Adaptation with Only Forward Passes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Forward-Optimization Adaptation (FOA), a backpropagation-free, weight-preserving test-time adaptation method that learns input prompts with CMA-ES under an unsupervised activation-statistics + entropy fitness, plus back-to-source activation shifting, enabling quantized ViT deployment.
Problem
Test-time adaptation is infeasible on resource-limited, quantized, or hard-coded deployed models because existing TTA methods rely on backpropagation and weight updates; the goal is online unsupervised adaptation to distribution shift without backprop or weight modification.
Approach
Learn only a small set of newly inserted input prompts online with a derivative-free CMA-ES optimizer, using an unsupervised fitness that combines prediction entropy with the discrepancy between test batch activation statistics and precomputed source statistics; additionally shift the final CLS activations from the test center toward the source center (back-to-source activation shifting).
Key results
- On ImageNet-C (level 5), FOA on full-precision ViT-Base reaches 66.3% average accuracy vs 59.6% for gradient-based TENT and 62.7% for SAR, with ECE 3.2%.
- FOA on 8-bit quantized ViT-Base achieves 63.5% accuracy on ImageNet-C, above TENT on 32-bit ViT (59.6%).
- Run-time memory at batch size 64 is 832 MB for FOA versus 5,165 MB for TENT and 16,836 MB for CoTTA; the 8-bit FOA uses 208 MB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace NoAdapt with FOA prompt adaptation and activation shifting | ViT-Base 32-bit · ImageNet-C level 5 | accuracy (%): 55.5 -> 66.3 | Helped |
| Use FOA instead of SAR | ViT-Base 32-bit · ImageNet-C level 5 | accuracy (%): 62.7 -> 66.3 | Helped |
| Use FOA instead of TENT | ViT-Base 8-bit versus ViT-Base 32-bit · ImageNet-C level 5 | accuracy (%): 59.6 -> 63.5 | Helped |
| Use FOA rather than TENT | ViT-Base 32-bit · ImageNet-R | accuracy (%): 63.9 -> 63.8 | No clear effect |
| Use FOA rather than T3A | ViT-Base 32-bit · ImageNet-V2 | accuracy (%): 75.5 -> 75.4 | No clear effect |
| Use FOA rather than CoTTA | ViT-Base 32-bit · ImageNet-Sketch | accuracy (%): 50.0 -> 49.9 | No clear effect |
| Add activation discrepancy to entropy-only CMA prompt adaptation (ablation) | FOA · ImageNet-C level 5 | accuracy (%): 44.9 -> 63.4 | Helped |
| Add activation shifting to activation discrepancy fitness (ablation) | FOA · ImageNet-C level 5 | accuracy (%): 63.4 -> 63.8 | Helped |
| Add activation shifting to entropy plus activation discrepancy fitness (ablation) | FOA · ImageNet-C level 5 | accuracy (%): 65.4 -> 66.3 | Helped |
| Add activation shifting alone instead of no adaptation (ablation) | FOA · ImageNet-C level 5 | accuracy (%): 55.5 -> 59.1 | Helped |
| Use entropy-only CMA fitness instead of no adaptation (ablation) | FOA · ImageNet-C level 5 | accuracy (%): 55.5 -> 44.9 | Hurt |
| Use interval updates every 4 samples rather than every 64 samples (ablation) | FOA-I single-sample adaptation · ImageNet-C Gaussian noise level 5 | accuracy (%): 61.5 -> 62.1 | Helped |
Surprising findings
- Directly replacing SGD with CMA for entropy-based TTA collapses accuracy (e.g., norm-layer CMA with entropy drops to 0.1%), showing entropy alone gives unreliable fitness signals.
- FOA performs worse than BN Adapt-style gains on ResNet-50 relative to TENT, indicating prompt-based forward-only adaptation transfers poorly to convolutional networks.
Limitations
- Requires K forward passes per batch (population size), so compute is higher than single-pass methods even though backprop is avoided.
- Weaker performance on convolutional networks (ResNet-50) because input prompts are hard to make location-invariant in CNNs.
- Needs batch statistics; single-sample adaptation requires an interval update variant (FOA-I).
Experimental setup
Datasets: ImageNet-1K ImageNet-C ImageNet-R ImageNet-V2 ImageNet-Sketch
Benchmarks: ImageNet-C ImageNet-R ImageNet-V2 ImageNet-Sketch
Metrics: accuracy expected calibration error run-time memory wall-clock time
Baselines: NoAdapt LAME T3A TENT SAR CoTTA MEMO
Builds on: CMA-ES TENT T3A LAME PTQ4ViT
Closest papers by meaning
- EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample (NeurIPS 2026 · similarity 0.91)
- Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment (NeurIPS 2025 · similarity 0.88)
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.87)
- Towards Real-World Test-Time Adaptation: Tri-net Self-Training with Balanced Normalization (AAAI 2024 · similarity 0.87)
- Bayesian Weight Enhancement with Steady-State Adaptation for Test-time Adaptation in Dynamic Environments (ICML 2025 · similarity 0.87)
- Backpropagation-free Network for 3D Test-time Adaptation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.