Research map / Architectures and efficiency / Attention and transformers
SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SATA, a training-free module that splits and merges ViT tokens by Moran's-I-style spatial autocorrelation scores before the FFN in later blocks, improving accuracy and corruption robustness of pretrained DeiT/ViT models.
Problem
Vision transformers remain vulnerable to corruptions and adversarial/natural shifts, and existing robustness methods need costly retraining or fine-tuning.
Approach
Applies local Moran's I spatial autocorrelation over token features (with attention map as the spatial weight matrix) to split tokens into groups in later blocks; only the middle-range tokens plus merged tokens go through the FFN, and residual tokens are concatenated back after the FFN, so no retraining is needed.
Key results
- SATA-B* (ViT-Base/16 + SATA) reaches 94.9% top-1 on ImageNet-1K without retraining, versus 82.0% for the DeiT-Base baseline in the ablation.
- SATA-B* achieves ImageNet-C mCE of 13.6, ImageNet-A top-1 of 63.6%, and ImageNet-R top-1 of 79.2%.
- SATA-B reduces GFLOPs to 15.9 from 17.6 for DeiT-B while reaching 93.9% top-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SATA to ViT-Base/16 | ViT-Base/16 · ImageNet-1K | Top-1 accuracy: 82.0 -> 94.9 | Helped |
| Add SATA to ViT-Base/16 | ViT-Base/16 · ImageNet-C | mCE: 48.5 -> 13.6 | Helped |
| Add SATA to ViT-Base/16 | ViT-Base/16 · ImageNet-A | Top-1 accuracy: 27.4 -> 63.6 | Helped |
| Add SATA to ViT-Base/16 | ViT-Base/16 · ImageNet-R | Top-1 accuracy: 44.9 -> 79.2 | Helped |
| Add SATA to ViT-Base/16 | ViT-Base/16 · ImageNet-Sketch | Top-1 accuracy: 32.4 -> 57.9 | Helped |
| Add token grouping without spatial-autocorrelation bounds (ablation) | SATA-B on ImageNet-1K · ImageNet-1K | Top-1 accuracy: 82.0 -> 84.4 | Helped |
| Add the upper bound and token grouping, without the lower bound (ablation) | SATA-B on ImageNet-1K · ImageNet-1K | Top-1 accuracy: 82.0 -> 92.8 | Helped |
| Add the lower bound and token grouping, without the upper bound (ablation) | SATA-B on ImageNet-1K · ImageNet-1K | Top-1 accuracy: 82.0 -> 92.5 | Helped |
| Add token grouping to token splitting with both bounds (ablation) | SATA-B on ImageNet-1K · ImageNet-1K | Top-1 accuracy: 92.3 -> 93.9 | Helped |
| Apply SATA from block 0.4 × B onward (ablation) | DeiT-Base/16 · ImageNet-1K | Top-1 accuracy | Helped |
| Vary α from 0.5 to 2, with α=1 as the selected setting (ablation) | SATA-B · ImageNet-1K | Top-1 accuracy | Mixed |
Surprising findings
- Spatial autocorrelation similarity between clean and corrupted inputs stays high across later blocks, while attention-map similarity drops significantly.
- Adversarial robustness appears unrelated to standard accuracy; Swin, PVT and TNT-S have higher clean accuracy than DeiT but much lower adversarial robustness.
Limitations
- Applied only to later blocks (gamma=0.7) and tuned on ImageNet-1K validation-style ablations; sensitive to alpha and gamma choices.
- Evaluated only on plain (non-window, non-hybrid) ViTs; extension to detection/segmentation and LLMs left to future work.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-C FGSM PGD
Metrics: top-1 accuracy top-5 accuracy mCE GFLOPs throughput latency
Baselines: DeiT-Ti DeiT-S DeiT-B RVT FAN RSPC TORA-ViT TransNeXt ConvNeXt-B EfficientNet-B4
Builds on: Token Merging (ToMe) Nikzad et al. spatial autocorrelation in CNNs Moran's I
Closest papers by meaning
- ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases (ICML 2021 · similarity 0.87)
- Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning (ICML 2025 · similarity 0.86)
- LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.85)
- Sub-token ViT Embedding via Stochastic Resonance Transformers (ICML 2024 · similarity 0.85)
- Sample- and Parameter-Efficient Auto-Regressive Image Models (CVPR 2025 · similarity 0.85)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.