Research map / Label-efficient and robust learning / Test-time adaptation
Asynchronous Perception Machine for Efficient Test Time Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Asynchronous Perception Machine (APM), a GLOM-inspired patch-wise, asynchronous architecture that performs test-time training by distilling a single CLIP/OpenCLIP representation per test image, with no augmentation or pretext task.
Problem
Test-time training (TTT) is computationally expensive because it requires repeated forward passes through large transformers, relies on surrogate pretext tasks such as augmentation or rotation, and needs parallel patch processing with high memory use.
Approach
A small network (a single conv filter producing a 'trigger column' plus a 5-layer shared MLP) is queried with each location-specific column (trigger column concatenated with sinusoidal positional encoding), so patches are processed one at a time; it is distilled once from a teacher CLS token and then overfit per test sample, with no augmentation or pretext task.
Key results
- On ImageNet-family OOD benchmarks with OpenCLIP ViT-H/14 teacher, APM averages 84.9% vs 81.4% for the teacher (ImageNet-A 84.2 vs 79.1); with CLIP ViT-B/16 it averages 62.6 vs 62.4 for TPT.
- APM reduces TTT compute from 462 GFLOPs (CLIP ViT-B/16 baseline) to 241.7 GFLOPs over 20 iterations, roughly a 50% reduction.
- On CIFAR-10-C (severity 5) APM reaches 14.8% average error vs 19.1% for TTT-Online and 24.5% for CLIP ViT-L/14.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APM instead of TPT with a CLIP-ViT-B/16 backbone | CLIP-ViT-B/16 test-time training · ImageNet natural-shift average | Top1 accuracy: 62.4 -> 62.6 | Helped |
| Use APM instead of CLIP VIT-L/14 | CLIP VIT-L/14 · ImageNet natural-shift average | Top1 accuracy: 72.5 -> 74.2 | Helped |
| Use APM instead of OpenCLIP-VIT-H/14 | OpenCLIP-VIT-H/14 · ImageNet natural-shift average | Top1 accuracy: 81.4 -> 84.9 | Helped |
| Use APM instead of OpenCLIP VIT-L/14 | OpenCLIP VIT-L/14 · ImageNet-C level 5 | Average accuracy: 42.1 -> 50.3 | Helped |
| Use APM instead of TPT | Cross-dataset generalization · Cross-dataset generalization average | Top-1 accuracy: 65.1 -> 65.5 | Helped |
| Use APM instead of TTT-Online | CIFAR-10-C test-time training · CIFAR-10-C level 5 | Average error rate: 19.1 -> 14.8 | Helped |
| Use APM instead of CLIP VIT-B/16 for 20 TTT iterations | CLIP VIT-B/16 TTT · 20 TTT iterations | GFlopsmeas: 462 -> 241.7 | Helped |
| Add RGB reconstruction loss (ablation) | APM with CLS loss · CIFAR-10 | Accuracy: 94.2 -> 91.0 | Hurt |
| Add feature-grid loss (ablation) | APM with CLS loss · CIFAR-10 | Accuracy: 94.2 -> 96.1 | Helped |
| Add RGB loss alongside CLS and feature-grid losses (ablation) | APM with CLS and feature-grid losses · CIFAR-10 | Accuracy: 96.1 -> 96.5 | Helped |
| Inject local patch Ixy into the trigger column (ablation) | APM with CLS, grid and RGB losses · CIFAR-10 | Accuracy: 96.5 -> 96.8 | Helped |
| Replace the convolutional trigger with routed ViT output tokens (ablation) | APM with local patch injection · CIFAR-10 | Accuracy: 96.8 -> 98.6 | Helped |
Surprising findings
- Data augmentation sharply hurts APM on CIFAR-10 test-time training (98.6% to 76.7%), contrary to standard TTT practice.
- A standard randomly initialized MLP achieves only 9.0% on CIFAR-10 when overfit on a single distilled sample, while APM reaches 97.04%.
Limitations
- Focused on image classification; dense tasks and video not evaluated.
- Requires multiple TTT iterations (about 20) and a teacher CLS token from a large pretrained model.
- Some results (e.g., ImageNet-C) compare to weaker baselines or use a teacher-dependent design; several reported findings are qualitative only.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-C CIFAR-10-C CIFAR-10 CIFAR-100 COCO DTD Flowers102
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-C CIFAR-10-C CIFAR-10 CIFAR-100 DTD Flowers102 Pets
Metrics: top-1 accuracy error rate GFLOPs memory (GB)
Baselines: TTT-Online TTT-MAE TPT CoOp CoCoOp CLIP ViT-B/16 CLIP ViT-L/14 OpenCLIP ViT-H/14
Builds on: GLOM TPT TTT-MAE CLIP neural fields
Closest papers by meaning
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models (ICML 2026 · similarity 0.83)
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.83)
- DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.83)
- Learning to (Learn at Test Time): RNNs with Expressive Hidden States (ICML 2025 · similarity 0.82)
- PerceptionGPT: Effectively Fusing Visual Perception into LLM (CVPR 2024 · similarity 0.82)
- DOTA: Distributional Test-time Adaptation of Vision-Language Models (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.