Research map / Architectures and efficiency / Attention and transformers
Vision Transformers provably learn spatial structure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use positional attention, with attention determined only by positional encodings, instead of vanilla attention | ViT-tiny-patch16-224 · ImageNet | test accuracy: 71.9% -> 68.9% | Hurt |
| Train on permuted rather than original CIFAR-100 images | ViT · CIFAR-100 | test accuracy | No clear effect |
| Train on permuted rather than original CIFAR-100 images | ResNet-18 and VGG-19 · CIFAR-100 | test accuracy | Mixed |
| Apply the inverse grid permutation to positional-encoding similarities | ViT trained on permuted CIFAR-100 · CIFAR-100 | recovery of spatially localized patterns | Helped |
| Train a one-layer ViT on spatially structured data with gradient descent | Simplified ViT with positional attention · Synthetic binary classification, idealized population-risk setting | generalization | Helped |
| Train with polynomially many samples and Gaussian random initialization | Simplified ViT with positional attention · Synthetic binary classification, empirical-risk setting | generalization | Helped |
| Pretrain patch association and keep the attention matrix fixed during downstream fine-tuning | Simplified ViT; fine-tune the value vector · Downstream data sharing structure but differing in features | test error | Helped |
| Use an algorithm without patch-association knowledge instead of the pretrained patch-association model | Binary classification algorithm · Downstream data sharing structure but differing in features | training samples needed for test error ≤ o(1) | Hurt |
| Use a generalized linear model without knowledge of the patch partition | Generalized linear model · Synthetic structured binary classification | classification error | Hurt |
| Remove spatial structure from the data distribution | One-layer ViT · Gaussian synthetic classification | patch association | Hurt |
Closest papers by meaning
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.90)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.89)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.89)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.89)
- Interpreting vision transformers via residual replacement model (NeurIPS 2025 · similarity 0.88)
- Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.