Research map / 3D vision / 3D generation
Multi-view Consistent 3D Panoptic Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVC-PSU, a TensoRF-based 3D panoptic radiance field that adds a probabilistic semantic aligner (PDC-Net correspondences) and geometric consistency enforcer (multi-view and monocular depth losses) to handle inconsistent 2D panoptic masks.
Problem
Panoptic segmentation masks generated by 2D models are multi-view inconsistent, causing ambiguities when lifting them into a 3D panoptic radiance field for novel view synthesis.
Approach
Align semantic predictions across views via dense-matching-based probabilistic averaging of machine-generated and rendered semantic logits (PSA), and enforce geometric consistency with multi-view reprojection depth loss plus a Pearson-correlated monocular depth prior from DPT (GCE).
Key results
- On ScanNet, MVC-PSU reaches 68.6 mIoU, 63.1 PQscene, 28.9 PSNR, versus Contrastive Lift 65.2/62.3/28.3.
- On Replica, MVC-PSU achieves 68.7 mIoU, 59.4 PQscene, 31.1 PSNR, outperforming Panoptic Lifting's 67.2/57.9/29.6.
- Ablation on ScanNet shows removing PSA drops mIoU by 2.3 and PQscene by 2.5; robustness to label noise up to about 50% is maintained.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Probabilistic Semantic Aligner (PSA) (ablation) | MVC-PSU · ScanNet | mIoU: 66.3 -> 68.6 | Helped |
| add Probabilistic Semantic Aligner (PSA) (ablation) | MVC-PSU · ScanNet | PQscene: 60.6 -> 63.1 | Helped |
| add Multi-view Depth Consistency (MVDC) (ablation) | MVC-PSU · ScanNet | mIoU: 68.1 -> 68.6 | Helped |
| add Multi-view Depth Consistency (MVDC) (ablation) | MVC-PSU · ScanNet | PSNR: 28.6 -> 28.9 | Helped |
| add Monocular Depth Constraint (MDC) (ablation) | MVC-PSU · ScanNet | mIoU: 67.9 -> 68.6 | Helped |
| add Monocular Depth Constraint (MDC) (ablation) | MVC-PSU · ScanNet | PSNR: 28.5 -> 28.9 | Helped |
| add both MVDC and MDC (ablation) | MVC-PSU · ScanNet | PSNR: 28.3 -> 28.9 | Helped |
| increase matching relations from 2 to 12 (ablation) | MVC-PSU · ScanNet | mIoU: 66.4 -> 68.8 | Helped |
| increase matching relations from 2 to 12 (ablation) | MVC-PSU · ScanNet | PQscene: 60.9 -> 63.2 | Helped |
| increase matching relations from 2 to 12 (ablation) | MVC-PSU · ScanNet | PSNR: 28.6 -> 28.9 | Helped |
| add PSA and GCE | Panoptic Lifting · ScanNet | PSNR: +0.6 | Helped |
| add PSA and GCE | Panoptic Lifting · Replica | PSNR: +1.5 | Helped |
Limitations
- Relies on dense matching quality; PSA requires enough texture overlap between views.
- Per-iteration training time increases with number of matching relations.
Experimental setup
Datasets: ScanNet Replica HyperSim
Benchmarks: ScanNet Replica HyperSim
Metrics: mIoU PQscene PSNR
Baselines: SemanticNeRF Mask2Former PNF DM-NeRF Panoptic Lifting Contrastive Lift
Builds on: Panoptic Lifting TensoRF Contrastive Lift PDC-Net
Closest papers by meaning
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.87)
- ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion (CVPR 2024 · similarity 0.87)
- How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation (ICML 2025 · similarity 0.87)
- Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception (CVPR 2024 · similarity 0.86)
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation (AAAI 2025 · similarity 0.86)
- PanoRecon: Real-Time Panoptic 3D Reconstruction from Monocular Video (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.