Research map / Architectures and efficiency / Attention and transformers
Steerable Transformers for Volumetric Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Steerable Transformers, an SE(d)-equivariant self-attention layer operating in Fourier space on top of steerable convolution encoders, and shows gains on Rotated MNIST, ModelNet10, PH2 and BraTS.
Problem
Standard transformers and attention are not equivariant to rotations and translations, and existing equivariant attention work for 3D focuses on point clouds, which is memory-heavy and poorly suited to dense volumetric grids.
Approach
Equivariant self-attention computed on steerable-convolution features in Fourier space, using steerable positional encodings (fixed spherical-harmonic basis times learnable inverse-square radial scalars) that satisfy the SE(d) steerability constraint, plus Fourier-space norm-based non-linearity and layer normalization.
Key results
- On Rotated MNIST, the steerable transformer (k=8) reaches 99.03% accuracy vs 98.97% for the steerable convolution baseline with fewer parameters (2.24M vs 2.54M).
- On BraTS, the steerable transformer improves dice scores to 75.01 / 54.89 / 76.37 (enhancing / core / whole tumor) vs 73.29 / 51.07 / 74.43 for the baseline.
- On ModelNet10 with SO(3) rotations, accuracy is 86.80% vs 86.62% for the steerable convolution baseline; gains are small and within about one standard deviation in several settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a two-block steerable transformer to the steerable convolution architecture | Steerable convolution classifier · Rotated MNIST (k=4) | accuracy: 98.72 ± 0.02 -> 98.82 ± 0.04 | Helped |
| Add a two-block steerable transformer to the steerable convolution architecture | Steerable convolution classifier · Rotated MNIST (k=8) | accuracy: 98.97 ± 0.01 -> 99.03 ± 0.04 | Helped |
| Add a two-block steerable transformer to the steerable convolution classifier | Steerable convolution classifier · ModelNet10 (z rotation) | accuracy: 90.13 ± 0.52 -> 90.40 ± 0.25 | Helped |
| Add a two-block steerable transformer to the steerable convolution classifier | Steerable convolution classifier · ModelNet10 (SO(3) rotation) | accuracy: 86.62 ± 0.25 -> 86.80 ± 0.58 | Helped |
| Replace the MLP between U-net paths with a two-block steerable transformer | Steerable convolution U-net · PH2 (k=4) | Dice score: 89.31 ± 0.17 -> 90.61 ± 0.90 | Helped |
| Replace the MLP between U-net paths with a two-block steerable transformer | Steerable convolution U-net · PH2 (k=8) | Dice score: 89.63 ±0.30 -> 90.72 ± 0.70 | Helped |
| Replace the MLP between U-net paths with a two-block steerable transformer | Steerable convolution U-net · BraTS (Enhancing Tumor) | Dice score: 73.29 ± 0.76 -> 75.01 ± 0.32 | Helped |
| Replace the MLP between U-net paths with a two-block steerable transformer | Steerable convolution U-net · BraTS (Tumor Core) | Dice score: 51.07 ± 1.23 -> 54.89 ± 0.76 | Helped |
| Replace the MLP between U-net paths with a two-block steerable transformer | Steerable convolution U-net · BraTS (Whole Tumor) | Dice score: 74.43 ± 0.77 -> 76.37 ± 0.46 | Helped |
Surprising findings
- Gains over the steerable convolution baseline are modest (often under one standard deviation), e.g. ModelNet10 z-rotation 90.40 vs 90.13.
Limitations
- Attention memory grows quickly with resolution, limiting network and batch size.
- Only modest Fourier cutoffs were used due to computational constraints.
- Improvements over steerable convolution baselines are small in several settings.
Experimental setup
Datasets: Rotated MNIST ModelNet10 PH2 BraTS
Benchmarks: Rotated MNIST ModelNet10 PH2 BraTS
Metrics: accuracy dice score parameter count runtime per batch
Baselines: Steerable Convolution baseline P4CNN Harmonic Net RotEqNet E2CNN alpha-R4 CNN GSA-Nets GE-ViT
Builds on: Vision Transformer steerable convolutional neural networks relative positional encoding Tensor Field Network SE(3)-transformer
Closest papers by meaning
- Steerable CNNs (ICLR 2017 · similarity 0.89)
- General E(2)-Equivariant Steerable CNNs (NeurIPS 2019 · similarity 0.88)
- CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers (NeurIPS 2025 · similarity 0.86)
- Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It (ICML 2025 · similarity 0.85)
- Clifford-Steerable Convolutional Neural Networks (ICML 2024 · similarity 0.84)
- A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.