Research map / Detection and segmentation / 3D detection and driving
SOGDet: Semantic-Occupancy Guided Multi-View 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOGDet, which adds a 3D semantic-occupancy branch to BEV-based multi-view 3D object detectors and fuses its features with the detection branch to improve NDS and mAP on nuScenes.
Problem
Multi-view BEV-based 3D object detectors ignore physical scene context (e.g., sidewalk, vegetation) that could help localize and classify objects.
Approach
Add a 3D semantic-occupancy (OC) branch alongside the object detection (OD) branch on a shared BEV feature, with a weighted-average modality-fusion module exchanging features between the two branches; two occupancy label types are explored (binary occupancy and semantic).
Key results
- SOGDet-SE-R101 reaches 56.6% NDS and 45.8% mAP on nuScenes val, versus 56.1% NDS and 44.9% mAP for AeDet-R101.
- SOGDet-SE improves BEVDet4D-depth R50 baseline from 49.0% to 50.6% NDS and 37.0% to 38.8% mAP on nuScenes val.
- On nuScenes test (R101), SOGDet-SE achieves 58.1% NDS and 47.4% mAP, above X3KD (56.1% NDS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic occupancy branch to BEVDet4D-depth | BEVDet4D-depth R50 · nuScenes val | mAP (%): 37.0 -> 38.8 | Helped |
| Add semantic occupancy branch to BEVDet4D-depth | BEVDet4D-depth R50 · nuScenes val | NDS (%): 49.0 -> 50.6 | Helped |
| Add semantic occupancy branch to BEVDet | BEVDet Tiny · nuScenes val | mAP (%): 31.2 -> 32.9 | Helped |
| Add semantic occupancy branch to BEVDet | BEVDet Tiny · nuScenes val | NDS (%): 39.2 -> 41.5 | Helped |
| Add semantic occupancy branch to BEVDet4D | BEVDet4D Tiny · nuScenes val | mAP (%): 33.8 -> 34.6 | Helped |
| Add semantic occupancy branch to BEVDet4D | BEVDet4D Tiny · nuScenes val | NDS (%): 47.6 -> 48.7 | Helped |
| Add semantic occupancy branch to BEVDepth | BEVDepth R50 · nuScenes val | mAP (%): 35.1 -> 37.2 | Helped |
| Add semantic occupancy branch to BEVDepth | BEVDepth R50 · nuScenes val | NDS (%): 47.5 -> 48.3 | Helped |
| Use binary occupancy labels instead of semantic labels (ablation) | SOGDet · nuScenes test, ResNet-101 | NDS (%): 58.1 -> 57.8 | Hurt |
| Use binary occupancy labels instead of semantic labels (ablation) | SOGDet · nuScenes test, ResNet-101 | mAP (%): 47.4 -> 47.1 | Hurt |
| Jointly train semantic occupancy with object detection | SOGDet-SE · nuScenes val, semantic occupancy | mIoU (%): 59.3 -> 58.6 | Hurt |
Surprising findings
- The occupancy branch alone achieves mIoU of 58.6% on nuScenes val, comparable to the dedicated occupancy method TPVFormer (59.3%), despite the simple design.
Limitations
- Occupancy branch uses a deliberately simple 3D-Conv head and does not follow standard occupancy labeling protocols.
- Binary occupancy results are only reported qualitatively.
- Evaluation limited to nuScenes; the paper itself notes future work with lidar and radar inputs.
Experimental setup
Datasets: nuScenes Panoptic nuScenes
Benchmarks: nuScenes validation set nuScenes test set
Metrics: NDS mAP mATE mASE mAOE mAVE mAAE mIoU
Baselines: BEVDet BEVDet4D BEVDepth AeDet BEVFormer PETR PETRv2 TPVFormer X3KD DETR3D
Builds on: BEVDet BEVDepth BEVDet4D CenterPoint TPVFormer
Closest papers by meaning
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.91)
- SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction (CVPR 2025 · similarity 0.89)
- OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.88)
- Towards Generalizable Multi-Camera 3D Object Detection via Perspective Rendering (AAAI 2025 · similarity 0.87)
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.87)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.