Research map / 3D vision / 3D generation
PanoRecon: Real-Time Panoptic 3D Reconstruction from Monocular Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces panoptic 3D reconstruction from posed monocular RGB video and PanoRecon, a real-time fragment-based system that jointly outputs geometry, semantics, and instances on ScanNetV2.
Problem
Holistic 3D scene understanding from posed monocular video requires jointly recovering surface geometry, semantic labels, and instance IDs online, which prior methods handle separately or require depth sensors for.
Approach
Depth-guided sparse back-projection of 2D features into fragment volumes from an off-the-shelf MVS network, a voxel-wise prediction network with TSDF, semantic, and center-offset heads, mean-shift voxel clustering for instances, and GNN+Sinkhorn matching with GRU fusion to track instances across fragments.
Key results
- Geometry F-score of 0.584 on ScanNetV2 test, above online feature fusion methods NeuRec (0.553) and Zuo et al. (0.572) and slightly above SimpleRecon (0.577).
- 3D semantic segmentation mIoU of 52.4 on ScanNetV2 test with RGB only, versus 34.0 for Atlas.
- 3D instance segmentation AP50 of 0.227 and AP25 of 0.484 with RGB only; runs at 12.85 keyframes per second (about 700 ms per 9-keyframe fragment).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use depth-guided feature projection (ablation) | PanoRecon · ScanNetV2 val set | F-score: 0.597 -> 0.631 | Helped |
| Use depth-guided feature projection (ablation) | PanoRecon · ScanNetV2 val set | mIoU: 53.6 -> 54.2 | Helped |
| Use 7 views per fragment instead of the default (ablation) | PanoRecon · ScanNetV2 val set | F-score: 0.631 -> 0.623 | No clear effect |
| Use 11 views per fragment instead of the default (ablation) | PanoRecon · ScanNetV2 val set | F-score: 0.631 -> 0.600 | Hurt |
| Replace EfficientNetV2-S with MnasNet (ablation) | PanoRecon · ScanNetV2 val set | F-score: 0.631 -> 0.596 | Hurt |
| Add TSDF volume augmentation (ablation) | PanoRecon · ScanNetV2 val set | F-score: 0.631 -> 0.624 | Hurt |
| Use depth sensor input (ablation) | PanoRecon · ScanNetV2 val set | F-score: 0.631 -> 0.810 | Helped |
Surprising findings
- Using ground-truth depth as input without retraining substantially improves geometry F-score (0.810 vs 0.631 in ablation), suggesting the method benefits from better depth models.
Limitations
- Relies on an external MVS depth network and SLAM poses, adding 430 ms of runtime per fragment.
- Instance segmentation AP is well below depth-based methods such as PBNet (AP50 0.747), and it uses no NMS post-processing.
- Online setting lacks global context, so it trails offline methods such as Atlas on some quality measures.
Experimental setup
Datasets: ScanNetV2
Benchmarks: ScanNetV2
Metrics: F-score accuracy completeness mIoU mAP@50 mAP@25 KFPS
Baselines: Atlas NeuralRecon Zuo et al. SimpleRecon Vortx MaskRCNN PanopticFusion PBNet
Builds on: NeuralRecon SimpleRecon PlanarRecon Atlas Sinkhorn algorithm
Closest papers by meaning
- Multi-view Consistent 3D Panoptic Scene Understanding (AAAI 2025 · similarity 0.86)
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.86)
- PanoWorld: Geometry-Consistent Panoramic Video World Modeling (NeurIPS 2026 · similarity 0.85)
- Scene4U: Hierarchical Layered 3D Scene Reconstruction from Single Panoramic Image for Your Immerse Exploration (CVPR 2025 · similarity 0.85)
- UniK3D: Universal Camera Monocular 3D Estimation (CVPR 2025 · similarity 0.85)
- Gated Fields: Learning Scene Reconstruction from Gated Videos (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.