Research map / Detection and segmentation / 3D detection and driving
PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PanoOcc, a camera-only framework that uses voxel queries and coarse-to-fine occupancy decoding to jointly perform 3D object detection and semantic/panoptic segmentation on nuScenes and dense occupancy on Occ3D.
Problem
Camera-based 3D scene understanding is fragmented across tasks (detection, BEV segmentation, occupancy); BEV features lose fine geometry needed for 3D panoptic segmentation, and dense 3D voxel representations are memory-expensive.
Approach
Learn a unified occupancy representation with voxel queries aggregating multi-frame, multi-view image features, decoded coarse-to-fine with optional occupancy sparsification, and jointly trained for detection and segmentation with voxel-selection feature propagation and a refine module producing panoptic output.
Key results
- Achieves 71.6 mIoU on nuScenes LiDAR semantic segmentation validation with R101-DCN and 74.5 mIoU with InternImage-XL, a new camera-based state of the art.
- Reaches 42.13 mIoU on Occ3D-nuScenes with camera mask training (R101-DCN), above BEVFormer* at 39.24.
- Achieves 62.1 PQ on nuScenes LiDAR panoptic segmentation validation using camera input only, approaching some LiDAR-based methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PanoOcc instead of OccFormer | Camera-based 3D semantic segmentation · nuScenes val | mIoU: 70.4 -> 71.6 | Helped |
| use PanoOcc instead of CTF-Occ | Camera-based occupancy prediction · Occ3D-nuScenes val | mIoU: 28.53 -> 32.47 | Helped |
| use PanoOcc* instead of BEVFormer* | Camera-based occupancy prediction with camera mask training · Occ3D-nuScenes val | mIoU: 39.24 -> 42.13 | Helped |
| jointly train detection and segmentation (ablation) | PanoOcc · nuScenes val | mIoU: 0.652 -> 0.656 | Helped |
| jointly train detection and segmentation (ablation) | PanoOcc · nuScenes val | mAP: 0.310 -> 0.266 | Hurt |
| add voxel selection for foreground voxels (ablation) | PanoOcc with joint detection and segmentation · nuScenes val | mIoU: 0.656 -> 0.661 | Helped |
| add voxel selection for foreground voxels (ablation) | PanoOcc with joint detection and segmentation · nuScenes val | mAP: 0.266 -> 0.271 | Helped |
| use 50x50x16 voxel queries with voxel upsampling instead of 200x200x8 queries (ablation) | PanoOcc · nuScenes val | mIoU: 67.9 -> 68.3 | Helped |
| use sparse rather than dense convolution for occupancy sparsify (ablation) | PanoOcc · nuScenes val | mIoU: 0.654 -> 0.639 | Hurt |
Surprising findings
- Coarse-to-fine decoding learns high-resolution semantic occupancy in a low-resolution latent space with no performance loss while halving training memory.
- Keeping only about 5% of voxels via occupancy sparsify causes no significant mIoU drop.
Limitations
- Panoptic quality lags LiDAR-based methods mainly due to weaker detection (48.4 vs 63.8 mAP).
- Occupancy sparsify is optional and evaluated only in a no-temporal R50 setting.
Experimental setup
Datasets: nuScenes Occ3D-nuScenes
Benchmarks: nuScenes Occ3D-nuScenes
Metrics: mIoU PQ PQ† mAP NDS RQ SQ FPS
Baselines: TPVFormer BEVFormer OccFormer MonoScene BEVDet CTF-Occ EfficientLPS Panoptic-PolarNet Panoptic-PHNet LidarMulitiNet RangeNet++ PolarNet
Builds on: BEVFormer TPVFormer DETR3D Focal loss Lovasz loss
Closest papers by meaning
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.89)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.88)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.88)
- ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction (AAAI 2025 · similarity 0.88)
- Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection (NeurIPS 2024 · similarity 0.88)
- ODG: Occupancy Prediction Using Dual Gaussians (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.