Research map / Architectures and efficiency / CNN design
Making Vision Transformers Truly Shift-Equivariant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Redesigns ViT tokenization, window self-attention, patch merging, and relative position embedding with data-adaptive, circularly shift-equivariant versions, yielding A-Swin, A-SwinV2, A-CvT, and A-MViTv2 with 100% circular shift consistency.
Problem
Vision Transformers are sensitive to small spatial shifts of the input; their modules (tokenization, window attention, patch merging, positional encoding) break shift-equivariance, so predictions change under shifts.
Approach
Make each ViT module input-adaptive: select the patch grid offset that maximizes a shift-invariant energy (ℓp norm) for tokenization and window partitioning, use APS+LPF striding for patch merging, and use a circular modular relative-distance table for RPE, giving provable circular shift-equivariance end to end.
Key results
- A-Swin-T, A-SwinV2-T, A-CvT-13 and A-MViTv2-T achieve 99.99-100% circular shift consistency on CIFAR-10/100 and ImageNet, versus 83-96% (C-Cons.) for the default models with their own accuracies.
- On ImageNet, A-Swin-T reaches 79.35% top-1 with 99.98% C-Cons. vs. Swin-T 78.5% and 86.68% C-Cons.
- On ADE20K, A-SwinV2 UperNet reaches 44.42 mIoU with 100% mASCC vs. SwinV2 UperNet 43.86 mIoU and 88.16 mASCC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive ViT modules instead of standard modules | Swin-T · CIFAR-10 circular shifts | Top-1 accuracy: 90.15 ± .18 -> 93.39 ± .12 | Helped |
| Use adaptive ViT modules instead of standard modules | Swin-T · CIFAR-10 circular shifts | C-Cons.: 83.30 ± .61 -> 99.99 ± .01 | Helped |
| Use adaptive ViT modules instead of standard modules | Swin-T · ImageNet circular shifts | Top-1 accuracy: 78.5 -> 79.35 | Helped |
| Use adaptive ViT modules instead of standard modules | Swin-T · ImageNet circular shifts | C-Cons.: 86.68 -> 99.98 | Helped |
| Use an adaptive ViT backbone | Swin-T + UperNet · ADE20K circular shifts | mIoU: 42.93 -> 43.44 | Helped |
| Use an adaptive ViT backbone | Swin-T + UperNet · ADE20K circular shifts | mASCC: 87.32 -> 100 | Helped |
| Add A-token (ablation) | A-Swin-T · CIFAR-10 circular shifts | Top-1 accuracy: 93.66 ± .19 -> 93.39 ± .13 | Hurt |
| Add A-token (ablation) | A-Swin-T · CIFAR-10 circular shifts | C-Cons.: 96.29 ± .20 -> 100 | Helped |
| Add A-WSA (ablation) | A-Swin-T · CIFAR-10 circular shifts | Top-1 accuracy: 93.24 ± .15 -> 93.39 ± .13 | Helped |
| Add A-WSA (ablation) | A-Swin-T · CIFAR-10 circular shifts | C-Cons.: 95.62 ± .54 -> 100 | Helped |
| Add A-PMerge (ablation) | A-Swin-T · CIFAR-10 circular shifts | Top-1 accuracy: 91.67 ± .10 -> 93.39 ± .13 | Helped |
| Add A-PMerge (ablation) | A-Swin-T · CIFAR-10 circular shifts | C-Cons.: 94.62 ± .11 -> 100 | Helped |
Surprising findings
- Adaptive tokenization without the other modules yields only 96.29% C-Cons. on CIFAR-10, showing all modules are needed for 100% consistency.
- Throughput drops by 10-20% for adaptive models, with A-Tokenization being the largest runtime cost (8.37 ms to 35.89 ms).
Limitations
- Perfect shift-equivariance holds only for circular shifts; under standard shifts consistency is not 100% due to boundary effects.
- Adaptive modules add 10-20% inference throughput overhead.
- CvT on ImageNet shows slightly lower accuracy under standard shifts than its baseline.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet ADE20K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet ADE20K
Metrics: top-1 accuracy circular shift consistency standard shift consistency mIoU mASCC mASSC throughput
Baselines: Swin-T SwinV2-T CvT-13 MViTv2-T UperNet with Swin-T UperNet with SwinV2-T anchoring method
Builds on: Swin Transformer SwinV2 CvT MViTv2 Adaptive Polyphase Sampling Learnable Polyphase Sampling
Closest papers by meaning
- Alias-Free ViT: Fractional Shift Invariance via Linear Attention (NeurIPS 2025 · similarity 0.88)
- Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers (AAAI 2025 · similarity 0.86)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.86)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.86)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.86)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.