Research map / Architectures and efficiency / Attention and transformers
Alias-Free ViT: Fractional Shift Invariance via Linear Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an Alias-Free ViT (AFT) built on XCiT with alias-free downsampling/activations/LayerNorm and shift-equivariant cross-covariance attention, achieving near-perfect fractional shift consistency while keeping ImageNet accuracy.
Problem
Vision transformers are not shift-invariant and are sensitive to small image translations, including sub-pixel (fractional) shifts, which prior anti-aliasing and APS methods for ViTs do not address.
Approach
Replace all non-shift-equivariant components of XCiT (patch embedding, LPI, MLP activations, LayerNorm, class token) with alias-free variants and use shift-equivariant linear/cross-covariance attention, so that the attention's K^T V term is shift-invariant and the output remains shift-equivariant; the class token is handled via an alias-free class attention yielding a shift-invariant representatio
Key results
- XCiT-Small-AF reaches 81.8% ImageNet top-1 vs 82.0% baseline, with 99.4% half-pixel shift consistency vs 89.8% for baseline.
- Under the half-pixel adversarial grid, XCiT-Small-AF keeps 81.1% adversarial accuracy vs 71.3% for baseline XCiT-Small.
- XCiT-Nano-AF attains 70.5% top-1 (vs 70.4% baseline) and 98.7% half-pixel consistency (vs 82.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard XCiT components with alias-free components | XCiT-Nano · ImageNet | half-pixel shift consistency: 82.0 -> 98.7 | Helped |
| replace standard XCiT components with alias-free components | XCiT-Small · ImageNet | adversarial half-pixel-grid accuracy: 71.3 -> 81.1 | Helped |
| replace standard XCiT components with alias-free components | XCiT-Nano · ImageNet | test accuracy: 70.4 -> 70.5 | No clear effect |
| replace standard XCiT components with alias-free components | XCiT-Small · ImageNet | test accuracy: 82.0 -> 81.8 | No clear effect |
| use APSPool for downsampling | XCiT-Nano · ImageNet | integer shift consistency: 83.7 -> 100.0 | Helped |
| use APSPool for downsampling | XCiT-Nano · ImageNet | half-pixel shift consistency: 82.0 -> 87.5 | Helped |
| use APSPool for downsampling | XCiT-Nano · ImageNet | test accuracy: 70.4 -> 68.4 | Hurt |
| use AF class attention instead of global average pooling (ablation) | XCiT-Small-AF · ImageNet | test accuracy: 80.70 -> 81.81 | Helped |
| use AF class attention instead of global average pooling (ablation) | XCiT-Nano-AF · ImageNet | test accuracy: 70.35 -> 70.48 | Helped |
| replace filtered GELU activations with polynomial activations throughout (ablation) | XCiT-Nano · ImageNet | test accuracy: 70.4 -> 65.8 | Hurt |
| replace filtered GELU activations with polynomial activations throughout (ablation) | XCiT-Nano · ImageNet | half-pixel shift consistency: 98.4 -> 99.6 | Helped |
| retain GELU in patch embedding and use polynomial activations in transformer blocks (ablation) | XCiT-Nano · ImageNet | test accuracy: 65.8 -> 68.5 | Mixed |
Surprising findings
- Replacing GELU with polynomial activations for shift-equivariance degrades accuracy substantially (65.8% vs 70.4%), unlike in prior convnet work.
- Removing positional encoding slightly improves accuracy in the hybrid XCiT-based design.
Limitations
- Substantial runtime increase from FFT-based alias-free downsampling and upsampling.
- Certified consistency holds only for cyclic shifts and sinc-interpolated fractional shifts, not natural-image translations.
- Polynomial activations for perfect invariance reduce accuracy significantly.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 Stanford Cars
Benchmarks: ImageNet CIFAR-10 CIFAR-100 Stanford Cars
Metrics: top-1 accuracy integer shift consistency half-pixel shift consistency adversarial accuracy
Baselines: XCiT XCiT-APS CvT-13 Swin-T ViT-Base
Builds on: XCiT Alias-Free GAN / alias-free activations Adaptive Polyphase Sampling (APS) Alias-Free ConvNet (AFC)
Closest papers by meaning
- Making Vision Transformers Truly Shift-Equivariant (CVPR 2024 · similarity 0.88)
- Improving Interpretation Faithfulness for Vision Transformers (ICML 2024 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.86)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.86)
- MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.