Research map / Architectures and efficiency / Attention and transformers
Stand-Alone Self-Attention in Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace spatial convolutions outside the stem with attention | ResNet-50 on ImageNet classification · ImageNet classification | accuracy (%): 76.9 -> 77.4 | Helped |
| Replace spatial convolutions, including the stem, with attention | ResNet-50 on ImageNet classification · ImageNet classification | accuracy (%): 76.9 -> 77.6 | Helped |
| Use attention in the backbone, FPN, and detection heads | RetinaNet · COCO object detection | mAP coco/50/75: 36.5 / 54.3 / 39.0 -> 36.6 / 54.5 / 39.2 | No clear effect |
| Keep early groups convolutional and use attention in later groups (ablation) | ResNet-50 on ImageNet classification · ImageNet classification, validation | Top-1 accuracy (%): 79.5 -> 80.7 | Helped |
| Use attention in early groups and convolution in later groups (ablation) | ResNet-50 on ImageNet classification · ImageNet classification, validation | Top-1 accuracy (%): 79.5 -> 79.7 | Hurt |
| Increase attention window from 3 × 3 to 9 × 9 (ablation) | ResNet-50 · ImageNet classification | Top-1 accuracy (%): 76.4 -> 77.7 | Helped |
| Use relative rather than absolute positional encoding (ablation) | Stand-alone attention · ImageNet classification, validation | Top-1 accuracy (%): 78.2 -> 80.2 | Helped |
| Add positional encoding rather than using none (ablation) | Stand-alone attention · ImageNet classification, validation | Top-1 accuracy (%): 77.6 -> 78.2 | Helped |
| Add content-content qᵀk interactions alongside qᵀr (ablation) | Stand-alone attention · ImageNet classification | Top-1 accuracy (%): 76.9 -> 77.4 | Helped |
| Use spatially aware value transformations in the attention stem (ablation) | ResNet-50 · ImageNet classification | Top-1 accuracy (%): 76.2 -> 77.6 | Helped |
| Use spatially aware value mixtures rather than a spatial convolution for stem values (ablation) | ResNet-50 · ImageNet classification | Top-1 accuracy (%): 77.2 -> 77.6 | Helped |
Closest papers by meaning
- Scaling Local Self-Attention for Parameter Efficient Visual Backbones (CVPR 2021 · similarity 0.93)
- Exploring Self-Attention for Image Recognition (CVPR 2020 · similarity 0.91)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.87)
- Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis (NeurIPS 2024 · similarity 0.87)
- Self-Attention-Based Contextual Modulation Improves Neural System Identification (ICLR 2025 · similarity 0.86)
- ResCLIP: Residual Attention for Training-free Dense Vision-language Inference (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.