Research map / Architectures and efficiency / Attention and transformers
Group Equivariant Stand-Alone Self-Attention For Vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rotation equivariance with R4 discretization | Self-attention on rotMNIST · rotMNIST | accuracy (%): 96.37 -> 97.46 | Helped |
| Increase rotation discretization from R4 to R8 | GSA-Net on rotMNIST · rotMNIST | accuracy (%): 97.46 -> 97.90 | Helped |
| Increase rotation discretization from R8 to R12 | GSA-Net on rotMNIST · rotMNIST | accuracy (%): 97.90 -> 97.97 | Helped |
| Increase rotation discretization from R12 to R16 | GSA-Net on rotMNIST · rotMNIST | accuracy (%): 97.97 -> 97.66 | Hurt |
| Add reflection equivariance | Self-attention on CIFAR10 · CIFAR10 | accuracy (%): 82.3 -> 83.72 | Helped |
| Add R4 rotation equivariance | Self-attention on PatchCamelyon · PatchCamelyon | accuracy (%): 83.04 -> 83.44 | Helped |
| Increase rotation discretization from R4 to R8 | GSA-Net on PatchCamelyon · PatchCamelyon | accuracy (%): 83.44 -> 83.58 | Helped |
| Add reflection equivariance to R4 rotation equivariance | GSA-Net on PatchCamelyon · PatchCamelyon | accuracy (%): 83.44 -> 84.76 | Helped |
| Vary neighborhood size from 3x3 to 5x5 (ablation) | R4_SA on rotMNIST · rotMNIST (early training stage) | accuracy (%): 96.56 -> 97.49 | Helped |
| Vary neighborhood size from 5x5 to 7x7 (ablation) | R4_SA on rotMNIST · rotMNIST (early training stage) | accuracy (%): 97.49 -> 97.33 | Hurt |
| Use larger neighborhoods rather than a 5x5 neighborhood (ablation) | R4_SA on rotMNIST · rotMNIST (early training stage) | accuracy (%): 97.49; 97.33; 97.42; 97.17; 96.89; 96.86; 97.05; 96.81 | Mixed |
| Use a smaller neighborhood instead of 5x5 (ablation) | R4_SA on rotMNIST · rotMNIST (early training stage) | accuracy (%): 97.49 -> 96.56 | Hurt |
Closest papers by meaning
- Attentive Group Equivariant Convolutional Networks (ICML 2020 · similarity 0.90)
- LieTransformer: Equivariant Self-Attention for Lie Groups (ICML 2021 · similarity 0.88)
- Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views? (ICLR 2022 · similarity 0.87)
- Probing Equivariance and Symmetry Breaking in Convolutional Networks (NeurIPS 2025 · similarity 0.86)
- Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions (NeurIPS 2025 · similarity 0.86)
- Variational Partial Group Convolutions for Input-Aware Partial Equivariance of Rotations and Color-Shifts (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.