Research map / 3D vision / 3D generation
PanoContext-Former: Panoramic Total Scene Understanding with a Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PanoContext-Former, a fully 3D transformer-based context module that jointly estimates panoramic room layout, oriented object boxes and object shapes, and releases the ReplicaPano dataset.
Problem
Holistic indoor scene understanding from a single 360-degree panorama, where layout estimation, object detection and shape reconstruction are usually solved separately with few cross-task correlations.
Approach
Lift the panorama to a point cloud and feed point, object, layout and global image features into a 6-layer transformer context module with random token masking and physical violation loss, so object-layout relationships are learned jointly; object shapes are predicted as latent codes from object features and decoded by a pretrained ONet.
Key results
- On iGibson-Synthetic, the method reaches 67.35 mAP over 11 categories versus 61.43 for DeepPanoContext-3D and 52.69 for DeepPanoContext.
- On ReplicaPano, object detection mAP is 49.02 versus 47.07 for Group-Free and 38.36 for DeepPanoContext-3D.
- Layout estimation on iGibson-Synthetic reaches 92.24 2D-IoU and 92.04 3D-IoU, above Deep3dLayout (90.65 / 90.40).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the 3D point cloud in DeepPanoContext | DeepPanoContext · iGibson-Synthetic | mAP: 52.69 -> 61.43 | Helped |
| Use the 3D point cloud in DeepPanoContext | DeepPanoContext · ReplicaPano | mAP: 26.41 -> 38.36 | Helped |
| Use ground-truth rather than estimated depth (ablation) | PanoContext-Former · iGibson-Synthetic | mAP: 69.09 -> 79.46 | Helped |
| Fine-tune the Panoformer depth estimator (ablation) | PanoContext-Former · iGibson-Synthetic | mAP: 54.77 -> 69.09 | Helped |
| Add perspective-view features (ablation) | PanoContext-Former · iGibson-Synthetic | 2D-IoU: 90.65 -> 90.98 | Helped |
| Add perspective-view features (ablation) | PanoContext-Former · iGibson-Synthetic | 3D-IoU: 90.40 -> 90.73 | Helped |
| Add joint training and physical violation loss (ablation) | PanoContext-Former · iGibson-Synthetic | mAP (11 categories): 62.59 -> 65.68 | Helped |
| Add joint training and physical violation loss (ablation) | PanoContext-Former · iGibson-Synthetic | 2D-IoU: 90.65 -> 91.41 | Helped |
| Add token masking (ablation) | PanoContext-Former · iGibson-Synthetic | mAP (11 categories): 65.68 -> 66.27 | Helped |
| Add a global image token (ablation) | PanoContext-Former · iGibson-Synthetic | mAP (11 categories): 65.68 -> 66.78 | Helped |
| Use token masking and a global image token together (ablation) | PanoContext-Former · iGibson-Synthetic | mAP (11 categories): 65.68 -> 67.35 | Helped |
| Use token masking and a global image token together (ablation) | PanoContext-Former · iGibson-Synthetic | 2D-IoU: 91.41 -> 92.24 | Helped |
Surprising findings
- DeepPanoContext-3D outperforms the proposed method on some categories such as floor lamp and chair, which the authors attribute to depth estimation failing on thin structures and to occlusion.
Limitations
- Depth estimation errors on thin structures (e.g., lamp poles) degrade detection performance.
- Heavily occluded objects are hard to detect without a 2D detector component.
- Object mesh generation is evaluated only qualitatively.
Experimental setup
Datasets: iGibson-Synthetic ReplicaPano ShapeNet Matterport3D NYU-37 labels
Benchmarks: iGibson-Synthetic ReplicaPano
Metrics: mAP (3D IoU 0.15) 2D-IoU 3D-IoU Abs.Rel. RMSE
Baselines: DeepPanoContext DeepPanoContext-3D Total3D IM3D Group-Free HorizonNet HoHoNet Led2Net Deep3DLayout
Builds on: Group-Free Deep3DLayout DeepPanoContext ONet Unifuse
Closest papers by meaning
- Geometric Exploitation for Indoor Panoramic Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
- PanoWorld: Towards Spatial Supersensing in 360◦ Panorama World (NeurIPS 2026 · similarity 0.88)
- HUSH: Holistic Panoramic 3D Scene Understanding using Spherical Harmonics (CVPR 2025 · similarity 0.87)
- OneCanvas: 3D Scene Understanding via Panoramic Reprojection (NeurIPS 2026 · similarity 0.86)
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.86)
- 360+x: A Panoptic Multi-modal Scene Understanding Dataset (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.