Research map / Architectures and efficiency / Attention and transformers
HEAL-SWIN: A Vision Transformer On The Sphere
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HEAL-SWIN, a Swin-transformer variant operating on the HEALPix spherical grid with grid and spiral shifting, evaluated on fisheye segmentation, depth estimation, and indoor spherical segmentation.
Problem
Applying CNNs and vision transformers to high-resolution wide-angle fisheye images suffers from projection and distortion losses when mapped to a flat rectangular grid, and existing spherical models struggle with resolution, uniformity, and partial-sphere coverage.
Approach
Use the nested, equal-area HEALPix grid to perform SWIN patching, windowing, patch merging/expansion and window shifting (grid-aligned or spiral along rings) directly on the sphere, using only the eight base pixels covering half the sphere.
Key results
- On Large SynWoodScape segmentation, HEAL-SWIN reaches 0.947 mIoU vs 0.918 for SWIN; on WoodScape 0.628 vs 0.617.
- On Stanford 2D-3D-S indoor segmentation, HEAL-SWIN achieves 44.3 mIoU, above the best spherical baseline HexRUNet at 43.3.
- Depth estimation on SynWoodScape yields lower Chamfer distance for HEAL-SWIN than SWIN, and HEAL-SWIN benefits more from larger training sets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HEAL-SWIN instead of flat SWIN | HEAL-SWIN · Large SynWoodScape | mIoU: 0.918 -> 0.947 | Helped |
| Use HEAL-SWIN instead of flat SWIN | HEAL-SWIN · Large+AD SynWoodScape | mIoU: 0.809 -> 0.841 | Helped |
| Use HEAL-SWIN instead of flat SWIN | HEAL-SWIN · WoodScape | mIoU: 0.617 -> 0.628 | Helped |
| Use HEAL-SWIN instead of flat SWIN | HEAL-SWIN · Stanford 2D-3D-S | mIoU: 43.3 -> 44.3 | Helped |
| Use HEAL-SWIN instead of flat SWIN | HEAL-SWIN · Stanford 2D-3D-S | mAcc: 58.6 -> 61.9 | Helped |
| Use HEAL-SWIN instead of flat SWIN for depth estimation | HEAL-SWIN · SynWoodScape | Chamfer distance | Helped |
| Increase the training set size (ablation) | HEAL-SWIN versus SWIN · Large+AD SynWoodScape | mIoU | Mixed |
| Use spiral shifting instead of grid shifting (ablation) | HEAL-SWIN · Semantic segmentation of indoor fisheye images | mIoU | Helped |
| Add an absolute position embedding after the patch embedding layer (ablation) | HEAL-SWIN · Image tasks | performance | No clear effect |
| Use window size 16, patch size 4, and shift size 2 with spiral shifting (ablation) | HEAL-SWIN · Semantic segmentation of indoor fisheye images | mIoU | Helped |
Surprising findings
- Spiral shifting, which introduces distortions toward the poles, slightly outperforms grid shifting in ablations.
- One HEAL-SWIN depth run with Chamfer distance 6.784 was excluded as an outlier via Chauvenet's criterion.
Limitations
- Grid is cut along base pixels, leaving parts of fisheye image uncovered and parts of the grid unused.
- Relative position bias does not account for differing base pixel geometry near poles and equator; model not rotation-equivariant.
- UNet-like decoder is not state-of-the-art for segmentation.
Experimental setup
Datasets: WoodScape SynWoodScape Stanford 2D-3D-S spherical MNIST
Benchmarks: WoodScape SynWoodScape Stanford 2D-3D-S spherical MNIST
Metrics: mIoU mAcc Chamfer distance inference time per pixel
Baselines: SWIN transformer Gauge CNN UGSCNN HexRUNet SphCNN Spin-SphCNN
Builds on: SWIN transformer HEALPix SWIN-V2 cosine attention UNet
Closest papers by meaning
- Attention on the Sphere (NeurIPS 2025 · similarity 0.83)
- TULIP: Transformer for Upsampling of LiDAR Point Clouds (CVPR 2024 · similarity 0.82)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.82)
- AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation (NeurIPS 2026 · similarity 0.82)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.82)
- Sharing Key Semantics in Transformer Makes Efficient Image Restoration (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.