Research map / Architectures and efficiency / Attention and transformers
SphereUFormer: A U-Shaped Transformer for Spherical 360 Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SphereUFormer, a U-shaped transformer operating directly on icosphere meshes with spherical local self-attention, up/downsampling and positional encodings, for 360° depth estimation and semantic segmentation.
Problem
Omnidirectional 360° perception suffers from distortions when images are projected to 2D grids (equirectangular or cube maps), while prior spherical-domain methods lag behind grid-based state of the art.
Approach
Builds a UFormer-style encoder-decoder directly on icosphere graphs, with spherical local self-attention over K-order neighbors using a shared learned relative position bias sampled from a 7x7 grid, icosphere-native down/up sampling by subdivision, and vertical-only absolute positional encoding to preserve horizontal rotation equivariance.
Key results
- On Structured3D at 256x512, the model reaches depth MAE 0.165 vs 0.169 for Elite360D (second best) and semantic mIoU 53.0 vs 52.0 for Elite360D.
- On Stanford2D3D at 512x1024, the model reaches MAE 0.147 and mIoU 71.5, versus 0.181 MAE and 63.3 mIoU for Elite360D.
- Removing relative positional encoding harms depth MAE most among ablations (0.251 vs 0.189 with full encoding).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SphereUFormer instead of the strongest reported baseline | 360° depth estimation · Stanford2D3D | MAE: .169 -> .165 | Helped |
| Use SphereUFormer instead of the strongest reported baseline | 360° depth estimation · Stanford2D3D | MRE: .069 -> .071 | Hurt |
| Use SphereUFormer instead of the strongest reported baseline | 360° depth estimation · Structured3D | MAE: .147 -> .142 | Helped |
| Use SphereUFormer instead of the strongest reported baseline | 360° semantic segmentation · Stanford2D3D | mIoU: 71.4 -> 72.2 | Helped |
| Use SphereUFormer instead of the strongest reported baseline | 360° semantic segmentation · Structured3D | mIoU: 52.0 -> 53.0 | Helped |
| Remove absolute and relative positional encodings (ablation) | SphereUFormer · Stanford2D3D depth estimation | MAE: .189 -> .326 | Hurt |
| Remove relative positional encoding (ablation) | SphereUFormer · Stanford2D3D depth estimation | MAE: .189 -> .251 | Hurt |
| Remove absolute positional encoding (ablation) | SphereUFormer · Stanford2D3D depth estimation | MAE: .189 -> .218 | Hurt |
| Set window coefficient Cwin to 0 (ablation) | SphereUFormer · Stanford2D3D depth estimation | MAE: .189 -> .412 | Hurt |
| Increase window coefficient Cwin from 1 to 2 (ablation) | SphereUFormer · Stanford2D3D depth estimation | MAE: .189 -> .175 | Helped |
| Increase window coefficient Cwin from 1 to 3 (ablation) | SphereUFormer · Stanford2D3D depth estimation | MAE: .189 -> .180 | Helped |
| Increase attention head coefficient Chead from 1 to 2 (ablation) | SphereUFormer · Stanford2D3D depth estimation | MAE: .189 -> .171 | Helped |
Surprising findings
- The model is about 30% slower than PanoFormer at runtime, due to unoptimized gather operations on the sphere graph, despite similar parameter and FLOP counts.
- Increasing the window coefficient beyond 2 did not improve performance, suggesting large windows interfere with convergence.
Limitations
- Runtime is about 30% slower than PanoFormer due to unoptimized sphere graph gather operations.
- Evaluated only on supervised depth estimation and semantic segmentation; generative and other tasks left unexplored.
- Patch-based baselines were not compared due to compute differences.
Experimental setup
Datasets: Stanford2D3D Structured3D
Benchmarks: Stanford2D3D Structured3D
Metrics: MAE MRE RMSE delta1 accuracy pixel accuracy mIoU
Baselines: PanoFormer EGFormer SFSS Elite360D HealSWIN HexRUnet
Builds on: UFormer PanoFormer
Closest papers by meaning
- Spherical Pseudo-Cylindrical Representation for Omnidirectional Image Super-resolution (AAAI 2024 · similarity 0.84)
- AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation (NeurIPS 2026 · similarity 0.84)
- SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model (AAAI 2024 · similarity 0.83)
- Learning Shape-Independent Transformation via Spherical Representations for Category-Level Object Pose Estimation (ICLR 2025 · similarity 0.83)
- Attention on the Sphere (NeurIPS 2025 · similarity 0.83)
- UniK3D: Universal Camera Monocular 3D Estimation (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.