Research map / 3D vision / 3D generation
PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Panoptic Scene Completion (PSC) for urban LiDAR and PaSCo, a MIMO-based sparse multi-scale mask-transformer that adds voxel- and instance-wise uncertainty via permutation-invariant mask ensembling.
Problem
Semantic scene completion from sparse urban LiDAR lacks instance-level predictions and uncertainty estimates, both needed for robotics and safety-critical driving.
Approach
Combines a multi-scale sparse generative U-Net with a Mask2Former-style transformer decoder on non-empty voxels, and uses MIMO multi-subnet training with a Hungarian-matched soft-IoU ensembling of unordered mask sets to yield voxel- and instance-wise uncertainty at near single-pass cost.
Key results
- On SemanticKITTI val, PaSCo reaches All PQ† 31.42 and PQ 16.51, versus 23.21 and 10.89 for the strongest baseline SCPNet* + MaskPLS.
- On SSCBench-KITTI360 test, PaSCo improves All PQ† by +8.09 and PQ by +3.45 over baselines.
- Compared with TTA, MC Dropout and Deep Ensemble under matched inference passes, PaSCo gives lower voxel ECE on SSCBench-KITTI360 (0.1348 vs 0.1540 for Deep Ensemble) at 1 pass versus 2-3 passes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PaSCo instead of the strongest baseline, SCPNet* + MaskPLS | panoptic scene completion · Semantic KITTI val | All PQ†: +8.21 | Helped |
| use PaSCo instead of the strongest baseline, SCPNet* + MaskPLS | panoptic scene completion · SSCBench-KITTI360 test | All PQ†: +8.09 | Helped |
| use PaSCo instead of SCPNet* + MaskPLS | panoptic scene completion · Semantic KITTI val | mIoU: +2.22 | Helped |
| use PaSCo instead of SCPNet* + MaskPLS | panoptic scene completion · SSCBench-KITTI360 test | mIoU: −0.27 | Hurt |
| use the MIMO ensemble instead of a single subnet (ablation) | PaSCo · Semantic KITTI val | All PQ†: 26.49 -> 31.42 | Helped |
| use the MIMO ensemble instead of a single subnet (ablation) | PaSCo · SSCBench-KITTI360 test | voxel ECE: 0.1749 -> 0.1348 | Helped |
| add inference augmentations (ablation) | PaSCo · Semantic KITTI val | All PQ†: 27.89 -> 31.42 | Helped |
| add rotation augmentation (ablation) | PaSCo · Semantic KITTI val | All PQ†: 28.84 -> 31.42 | Helped |
| add voxel-query semantic loss (ablation) | PaSCo · Semantic KITTI val | All PQ†: 28.82 -> 31.42 | Helped |
| use semantic pruning instead of binary occupancy pruning (ablation) | PaSCo · Semantic KITTI val | All PQ†: 30.12 -> 31.42 | Helped |
| use soft matching instead of hard binary-mask IoU matching (ablation) | PaSCo mask ensembling · Semantic KITTI | All PQ†: -3.75 | Helped |
| increase subnet count from 1 to 3 (ablation) | PaSCo · Semantic KITTI val | All PQ†: 26.49 -> 31.42 | Helped |
Surprising findings
- PSC and SSC metrics are not directly correlated: PaSCo greatly improves PQ while mIoU is on par with baselines (second on SSCBench-KITTI360).
- Averaging subnets peaks at M=2, while the proposed ensembling still improves at M=3.
Limitations
- Limited number of subnets: performance degrades as subnets increase because per-subnet capacity drops.
- May overlook or merge small, semantically similar nearby objects.
- Does not distinguish epistemic from aleatoric uncertainty.
Experimental setup
Datasets: SemanticKITTI SSCBench-KITTI360 Robo3D (SemanticKITTI-C)
Benchmarks: SemanticKITTI SSCBench-KITTI360 Robo3D SemanticKITTI-C
Metrics: PQ PQ† SQ RQ mIoU ECE NLL
Baselines: LMSCNet + MaskPLS JS3CNet + MaskPLS SCPNet + MaskPLS SCPNet* + MaskPLS TTA MC Dropout Deep Ensemble
Builds on: MIMO Mask2Former MaskPLS SCPNet Deep Ensemble
Closest papers by meaning
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.90)
- Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion (CVPR 2025 · similarity 0.87)
- Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance (NeurIPS 2025 · similarity 0.86)
- Memory-Augmented Re-Completion for 3D Semantic Scene Completion (AAAI 2025 · similarity 0.86)
- VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion (AAAI 2025 · similarity 0.86)
- Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.