Research map / Architectures and efficiency / Attention and transformers
Attention on the Sphere
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes attention to the 2-sphere via quadrature-weighted global and geodesic-neighborhood attention, building spherical ViT/SegFormer variants that outperform Euclidean counterparts on shallow water, segmentation, and depth tasks.
Problem
Standard Transformers operate on planar grids and do not respect the rotational symmetry and topology of spherical data, and Euclidean attention on equirectangular grids gives distorted neighborhoods and poor long rollout stability.
Approach
Derives a continuous attention formulation on S^2 using the Haar measure, then discretizes it with quadrature weights (incorporated into the softmax denominator and numerator) for global attention, and uses geodesic-disk indicator masks for neighborhood attention, yielding approximate SO(3) equivariance and neural-operator behaviour.
Key results
- On the shallow water equations, the S^2 local Transformer reduces validation loss to 0.007 from 0.058 for the R^2 local Transformer.
- On 128x256 Stanford 2D3DS segmentation, the S^2 SegFormer (local) reaches IoU 0.645 vs 0.623 for the R^2 SegFormer (local) and accuracy 0.968 vs 0.965.
- On depth estimation, the S^2 Transformer (local) achieves L1 error 0.191 vs 0.223 for the R^2 Transformer (local).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use an S² SegFormer with local attention instead of the R² local Transformer | Stanford 2D3DS segmentation, 128×256 · Stanford 2D3DS segmentation, 128×256 | IoU: 0.630\pm 0.140 -> 0.645\pm 0.132 | Helped |
| use an S² local SegFormer instead of an R² local SegFormer | Stanford 2D3DS segmentation, 256×512 · Stanford 2D3DS segmentation, 256×512 | IoU: 0.692\pm 0.158 -> 0.728\pm 0.136 | Helped |
| use an S² local SegFormer instead of an R² SegFormer | Stanford 2D3DS depth estimation · Stanford 2D3DS depth estimation | Validation loss: 1.239\pm0.367 -> 1.182\pm0.390 | Helped |
| use an S² local Transformer instead of an R² local Transformer | Shallow water equations on rotating sphere · Shallow water equations on rotating sphere | Validation loss: 0.058\pm 0.003 -> 0.007\pm 0.000 | Helped |
| replace patch embeddings with an R² convolutional encoder (ablation) | ViT · Shallow water equations on rotating sphere | Validation loss: 0.826\pm0.026 -> 0.321\pm0.015 | Helped |
| replace learned positional embeddings with spectral positional embeddings (ablation) | ViT with R² convolutional encoder · Shallow water equations on rotating sphere | Validation loss: 0.321\pm0.015 -> 0.338\pm0.013 | Hurt |
| replace the R² convolutional encoder with an S² convolutional encoder (ablation) | R² Transformer with spectral positional embeddings · Shallow water equations on rotating sphere | Validation loss: 0.338\pm0.013 -> 0.036\pm0.002 | Helped |
| replace R² attention with S² attention (ablation) | S² Transformer with S² convolutional encoder · Shallow water equations on rotating sphere | Validation loss: 0.036\pm0.002 -> 0.034\pm0.001 | Helped |
| replace S² global attention with S² local attention (ablation) | S² Transformer with S² attention · Shallow water equations on rotating sphere | Validation loss: 0.034\pm0.001 -> 0.007\pm0.000 | Helped |
Surprising findings
- None of the tested architectures remain stable in autoregressive rollouts for the shallow water equations despite strong single-step performance.
- On segmentation, the R2 SegFormer (local) and S2 SegFormer (local) differ only slightly in accuracy, and on depth the R2 SegFormer has lower L1 error than the spherical SegFormer.
Limitations
- Custom CUDA neighborhood attention kernels are slower than PyTorch SDPA and do not use reduced precision or tensor cores.
- Models do not show stability in autoregressive rollouts for shallow water equations.
- Distributed memory support for spherical attention is not yet implemented.
Experimental setup
Datasets: Stanford 2D3DS shallow water equations simulated data
Benchmarks: Stanford 2D3DS shallow water equations on rotating sphere
Metrics: IoU accuracy cross entropy loss L1 error L2 error W1,1 error validation loss
Baselines: ViT SegFormer R2 Transformer (local) SphereUFormer Euclidean SegFormer (local)
Builds on: Vision Transformer SegFormer Neighborhood Transformer DISCO discrete-continuous convolutions NATTEN
Closest papers by meaning
- Spherical SO(3) Equivariant Local Attention (ICML 2026 · similarity 0.88)
- Unified Spherical Frontend: Learning Rotation-Equivariant Representations of Spherical Images from Any Camera (CVPR 2026 · similarity 0.87)
- Spin-Weighted Spherical CNNs (NeurIPS 2020 · similarity 0.87)
- Efficient Generalized Spherical CNNs (ICLR 2021 · similarity 0.87)
- GeoTransolver: Learning Physics on Irregular Domains using Multi-scale Geometry Aware Physics Attention Transformer (NeurIPS 2026 · similarity 0.85)
- Linear Spherical Sliced Optimal Transport: A Fast Metric for Comparing Spherical Data (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.