Research map / 3D vision / 3D generation
Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GSD-Occ, a real-time vision-based 3D occupancy predictor that uses a dual-branch geometry/semantic network with re-parameterized large-kernel 3D convolution and a mixup-of-GT-and-predicted-depth learning strategy, reaching 39.4% mIoU at 20 FPS on Occ3D-nuScenes.
Problem
Vision-based 3D occupancy prediction for autonomous driving is accurate but computationally expensive and slow (1-3 FPS, >10 GB memory), which conflicts with real-time deployment needs; geometry and semantics are strongly coupled through depth-guided view transformation.
Approach
Disentangle geometry and semantics both architecturally (a voxel branch with re-parameterized large-kernel 3D convolution for geometry and a BEV branch with BEV-level temporal fusion for semantics) and in training (gradually mixing ground-truth depth with predicted depth in LSS via a sigmoid schedule).
Key results
- GSD-Occ achieves 39.4% mIoU at 20.0 FPS on Occ3D-nuScenes validation with 4759 MB memory, versus 37.5% mIoU for FB-OCC at 7.0 FPS.
- GSD-Occ reaches 38.9 RayIoU at 20 FPS, 4.9 points above SparseOcc (34.0) at 17.7 FPS.
- Geometric-semantic disentangled learning adds 0.3-0.55 mIoU to different pretrained models without extra computation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GSD-Occ instead of FastOCC | Occ3D-nuScenes val, real-time methods · Occ3D-nuScenes val | mIoU: 34.2 -> 39.4 | Helped |
| Use GSD-Occ instead of SparseOcc | Occ3D-nuScenes val, no visible mask · Occ3D-nuScenes val, no visible mask | mIoU: 30.6 -> 31.8 | Helped |
| Use GSD-Occ instead of SparseOcc | Occ3D-nuScenes val · Occ3D-nuScenes val | RayIoU: 34.0 -> 38.9 | Helped |
| Add the 2D semantic encoder (ablation) | GSDBN · Occ3D-nuScenes val | mIoU: 35.11 -> 36.38 | Helped |
| Use voxel features instead of BEV features for temporal fusion (ablation) | GSDBN · Occ3D-nuScenes val | mIoU: 38.40 -> 39.09 | Helped |
| Add RLK-3DConv (ablation) | GSDBN · Occ3D-nuScenes val | mIoU: 38.40 -> 38.90 | Helped |
| Replace RLK-3DConv with non-re-parameterized large-kernel 3D convolution (ablation) | GSDBN · Occ3D-nuScenes val | mIoU: 38.90 -> 38.10 | Hurt |
| Add GSDL (ablation) | FB-OCC with BEVDepth pretraining · Occ3D-nuScenes val | mIoU: +0.32 | Helped |
| Add GSDL (ablation) | GSD-Occ with ImageNet pretraining · Occ3D-nuScenes val | mIoU: +0.40 | Helped |
| Add GSDL (ablation) | GSD-Occ with BEVDepth pretraining · Occ3D-nuScenes val | mIoU: +0.55 | Helped |
| Replace Channel-to-Height lifting with BVL (ablation) | GSD-Occ · Occ3D-nuScenes val | mIoU: 38.62 -> 39.45 | Helped |
| Replace Repeat + 3D Conv lifting with BVL (ablation) | GSD-Occ · Occ3D-nuScenes val | mIoU: 38.42 -> 39.45 | Helped |
Surprising findings
- Directly training with ground-truth depth and testing with predicted depth yields only 35.1% mIoU, showing the model fails to learn to refine predicted geometry.
- Plain large-kernel 3D convolution (LK-3DConv) degrades accuracy relative to the re-parameterized multi-branch version, suggesting that simply enlarging the receptive field can harm features.
Limitations
- Real-time claims depend on A100 timing and a 256x704 input; the best 41.7 mIoU variant runs at 10 FPS.
- Results are reported only on Occ3D-nuScenes validation with a ResNet-50 backbone.
Experimental setup
Datasets: Occ3D-nuScenes nuScenes ImageNet
Benchmarks: Occ3D-nuScenes
Metrics: mIoU RayIoU FPS memory (MB)
Baselines: FB-OCC FastOcc SparseOcc PanoOcc COTR BEVFormer BEVDet4D SurroundOcc FlashOcc BEVoxSeg
Builds on: LSS FB-OCC RepVGG RepLKNet BEVDepth
Closest papers by meaning
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.92)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.92)
- COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction (CVPR 2024 · similarity 0.91)
- Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles (CVPR 2024 · similarity 0.91)
- SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction (CVPR 2025 · similarity 0.90)
- Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.