Research map / 3D vision / 3D generation
PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PanoGS, a 3DGS-based method for 3D panoptic open-vocabulary scene understanding that learns 3D language features from a latent pyramid tri-plane and groups Gaussians into instances via language-guided graph cuts and SAM-guided graph clustering.
Problem
Existing 3DGS open-vocabulary methods predict per-query heatmaps over scene features and cannot distinguish 3D instances of the same semantic class, so they cannot perform panoptic segmentation; they also learn lossy or noisy per-primitive language features.
Approach
Model language features with a latent pyramid tri-plane decoded by a 3D feature decoder and distilled from confidence-weighted multi-view fused features in 3D space; group Gaussians into super-primitives with language-guided graph cuts, then cluster them progressively using Jensen-Shannon divergence of SAM mask-label distributions across views.
Key results
- On ScanNetV2, PanoGS reaches 50.72 mIoU and 70.20 mAcc for open-vocabulary 3D semantic segmentation, above OpenScene (Ens.) at 47.63 mIoU.
- On Replica, PanoGS reaches 54.98 mIoU and 67.35 mAcc, versus 49.03 mIoU for OpenScene (Ens.).
- Ablation on Replica shows multi-view JSD affinity raises PRQ (T) from 20.28 to 33.22 over feature-similarity affinity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PanoGS instead of OpenScene (Ens.) | 3D semantic segmentation · ScanNetV2 | mIoU: 47.63 -> 50.72 | Helped |
| Use PanoGS instead of OpenScene (Ens.) | 3D semantic segmentation · ScanNetV2 | mAcc.: 69.74 -> 70.20 | Helped |
| Use PanoGS instead of OpenScene (Ens.) + supervised masks | 3D panoptic segmentation · ScanNetV2 | PRQ (T): 43.77 -> 49.26 | Hurt |
| Use PanoGS instead of OpenScene (Dis.) + supervised masks | 3D panoptic segmentation · ScanNetV2 | PRQ (S): 40.69 -> 48.24 | Helped |
| Use PanoGS instead of OpenScene (Ens.) | 3D semantic segmentation · Replica | mIoU: 49.03 -> 54.98 | Helped |
| Use PanoGS instead of OpenScene (Ens.) | 3D semantic segmentation · Replica | mAcc.: 62.89 -> 67.35 | Helped |
| Use PanoGS instead of OpenScene (Ens.) + supervised masks | 3D panoptic segmentation · Replica | PRQ (T): 33.04 -> 43.04 | Helped |
| Use PanoGS instead of OpenScene (Ens.) + supervised masks | 3D panoptic segmentation · Replica | PRQ (S): 11.84 -> 30.60 | Helped |
| Add the 3D decoder and pyramid tri-plane (ablation) | PanoGS · ScanNetV2 | mIoU: 24.51 -> 50.72 | Helped |
| Add the 3D decoder and pyramid tri-plane (ablation) | PanoGS · ScanNetV2 | PRQ (T): 20.57 -> 33.84 | Helped |
| Add the pyramid tri-plane rather than positional encoding (ablation) | PanoGS with 3D decoder · ScanNetV2 | mIoU: 36.39 -> 50.72 | Helped |
| Add multi-view JSD mask-label affinity (ablation) | PanoGS graph clustering · Replica | PRQ (T): 20.28 -> 33.22 | Helped |
Surprising findings
- With SoftGroup instance proposals, PanoGS's thing PRQ on ScanNetV2 (33.84) is lower than the OpenScene+SoftGroup combination (43.77), though its own semantic mIoU is higher.
Limitations
- Panoptic PRQ on ScanNetV2 without supervised SoftGroup proposals is lower than OpenScene+SoftGroup for thing classes.
- Baselines produce only point-level outputs, so panoptic comparison requires a supervised SoftGroup instance model trained on ScanNetV2.
Experimental setup
Datasets: Replica ScanNetV2
Benchmarks: ScanNetV2 Replica
Metrics: mIoU mAcc PRQ (T) PRQ (S)
Baselines: LangSplat OpenGaussian OpenScene SoftGroup
Builds on: 3D Gaussian Splatting LangSplat OpenScene SAM LSeg
Closest papers by meaning
- OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding (NeurIPS 2024 · similarity 0.91)
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting (NeurIPS 2025 · similarity 0.90)
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.90)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.89)
- ReferSplat: Referring Segmentation in 3D Gaussian Splatting (ICML 2025 · similarity 0.89)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.