Research map / Architectures and efficiency / Attention and transformers
Exploring Self-Attention for Image Recognition
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace convolution with pairwise self-attention | ResNet-style ImageNet classifier · ImageNet val-original | top-1 accuracy: 73.6 -> 74.9; 76.0 -> 76.6; 76.9 -> 76.9 | Helped |
| Replace convolution with patchwise self-attention | ResNet-style ImageNet classifier · ImageNet val-original | top-1 accuracy: 73.6 -> 77.1; 76.0 -> 78.0; 76.9 -> 78.2 | Helped |
| Use vector-valued relations rather than scalar dot-product attention (ablation) | SAN10-pairwise · ImageNet val-split | top-1 accuracy: 77.0 -> 77.4 | Helped |
| Use concatenation instead of subtraction as the pairwise relation (ablation) | SAN10-pairwise · ImageNet val-split | top-1 accuracy: 77.4 -> 76.4 | Hurt |
| Use concatenation instead of star-product for patchwise relations (ablation) | SAN10-patchwise · ImageNet val-split | top-1 accuracy: 78.7 -> 79.3 | Helped |
| Use two-layer rather than one-layer attention mapping (ablation) | SAN10-pairwise · ImageNet val-split | top-1 accuracy: 75.8 -> 77.4 | Helped |
| Use two-layer rather than one-layer attention mapping (ablation) | SAN10-patchwise · ImageNet val-split | top-1 accuracy: 79.3 -> 79.3 | No clear effect |
| Keep φ, ψ, and β transformations distinct (ablation) | SAN10-pairwise · ImageNet val-split | top-1 accuracy: 76.5 -> 77.4 | Helped |
| Keep φ, ψ, and β transformations distinct (ablation) | SAN10-patchwise · ImageNet val-split | top-1 accuracy: 78.9 -> 79.3 | Helped |
| Increase pairwise footprint from 3×3 to 9×9 (ablation) | SAN10-pairwise · ImageNet val-split | top-1 accuracy: 75.3 -> 77.8 | Helped |
| Increase patchwise footprint from 3×3 to 7×7 (ablation) | SAN10-patchwise · ImageNet val-split | top-1 accuracy: 77.4 -> 79.3 | Helped |
| Use relative instead of no position encoding (ablation) | SAN10-pairwise · ImageNet val-split | top-1 accuracy: 72.3 -> 77.4 | Helped |
Closest papers by meaning
- On the Integration of Self-Attention and Convolution (CVPR 2022 · similarity 0.92)
- Stand-Alone Self-Attention in Vision Models (NeurIPS 2019 · similarity 0.91)
- Scaling Local Self-Attention for Parameter Efficient Visual Backbones (CVPR 2021 · similarity 0.91)
- Explicitly Modeled Attention Maps for Image Classification (AAAI 2021 · similarity 0.91)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.87)
- Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.