Research map / 3D vision / 3D generation
ProtoOcc: Accurate, Efficient 3D Occupancy Prediction Using Dual Branch Encoder-Prototype Query Decoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProtoOcc, a vision-based 3D occupancy predictor combining a dual-branch voxel/BEV encoder with a prototype-query decoder that predicts occupancy in one pass, plus noise-based robust prototype learning.
Problem
Vision-based 3D occupancy prediction trades off accuracy against efficiency: voxel 3D CNNs are costly and have small receptive fields, BEV loses height detail, and query-based decoders need multiple iterative layers that slow inference.
Approach
A Dual Branch Encoder fuses small-kernel 3D voxel features with large-kernel 2D BEV features across scales, and a Prototype Query Decoder builds class queries from scene-adaptive (mask-pooled) and scene-agnostic (EMA) prototypes so occupancy is predicted in a single step without iterative Transformer decoding; Robust Prototype Learning injects mask noise during training.
Key results
- Single-frame ProtoOcc reaches 39.56% mIoU on Occ3D-nuScenes val at 77.9 ms (about 12.83 FPS) on an RTX 3090, outperforming FB-Occ (37.39%) and COTR (37.02%).
- Multi-frame (8 frames) ProtoOcc reaches 45.02% mIoU, surpassing COTR's 44.45%.
- Ablation: DBE adds +1.69 mIoU, PQD adds +1.45 mIoU at similar latency, and RPL adds +0.4 mIoU on the 1/4-data training setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the baseline encoder with DBE (ablation) | ProtoOcc baseline · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: +1.69 | Helped |
| Replace the baseline decoder with PQD (ablation) | ProtoOcc baseline · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: +1.45 | Helped |
| Add DBE and PQD to the baseline (ablation) | ProtoOcc baseline · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: +2.87 | Helped |
| Add RPL to DBE + PQD (ablation) | ProtoOcc · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: +0.4% | Helped |
| Increase voxel-branch kernel size from 3 to 7 (ablation) | Voxel-only branch · Occ3D-nuScenes validation, 1/4 training data ablation | Latency (ms): 60.7 -> 87.2 | Hurt |
| Increase BEV-branch kernel size from 3 to 7 (ablation) | BEV-only branch · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: 35.64 -> 36.07 | Helped |
| Add multi-scale fusion to dual branch with kernels 3/3 (ablation) | Dual Branch Encoder · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: 36.70 -> 36.93 (+0.23%) | Helped |
| Use BEV kernel size 7 instead of 3, keeping voxel kernel size 3 and multi-scale fusion (ablation) | Dual Branch Encoder · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: 36.93 -> 37.45 (+0.52%) | Helped |
| Replace iterative query decoding with AdaPG and AgnoPG prototypes (ablation) | ProtoOcc decoder · Occ3D-nuScenes validation, 1/4 training data ablation | Latency (ms): 151.6 -> 77.9 | Helped |
| Add AgnoPG to AdaPG (ablation) | PQD · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: 36.87 -> 37.45 | Helped |
| Use average pooling instead of sum or max pooling for prototype generation (ablation) | AdaPG · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: 37.15; 37.35; 37.45 | Helped |
| Set EMA coefficient alpha to 0.01 (ablation) | AgnoPG · Occ3D-nuScenes validation, 1/4 training data ablation | mIoU: 37.35; 37.45; 37.33; 37.28 | Helped |
Surprising findings
- Prototype-based single-step decoding outperforms the iterative query-based decoder (36.87 vs 36.66 mIoU in the decoder comparison) while cutting latency roughly in half (77.4 ms vs 151.6 ms).
Limitations
- Gains from individual components are modest (0.4 to 2.9 mIoU points), and ablations use a quarter of the training data.
- Multi-frame results require fusing eight frames, which adds cost not reflected in the single-frame latency.
- Evaluation focuses on the Occ3D-nuScenes benchmark plus one additional SemanticKITTI comparison.
Experimental setup
Datasets: Occ3D-nuScenes nuScenes SemanticKITTI
Benchmarks: Occ3D-nuScenes SemanticKITTI
Metrics: mIoU latency (ms) FPS
Baselines: MonoScene TPVFormer Vampire CTF-Occ SurroundOcc BEVDet OccFormer COTR FB-Occ FastOcc PanoOcc BEVFormer
Builds on: Lift-Splat-Shoot OccFormer FB-Occ BEVDet4D LargeKernel3D
Closest papers by meaning
- 3D Occupancy Prediction with Low-Resolution Queries via Prototype-aware View Transformation (CVPR 2025 · similarity 0.95)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.90)
- AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks (NeurIPS 2026 · similarity 0.89)
- OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries (NeurIPS 2024 · similarity 0.88)
- LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-based 3D Semantic Occupancy Prediction (CVPR 2024 · similarity 0.88)
- ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.