Research map / 3D vision / 3D generation
3D Occupancy Prediction with Low-Resolution Queries via Prototype-aware View Transformation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProtoOcc, a camera-based 3D occupancy network that maps clustered 2D image prototypes onto low-resolution voxel queries and decodes them with multi-perspective voxel augmentations plus consistency regularization.
Problem
Camera-based 3D occupancy prediction loses visual detail when voxel query resolution is reduced for real-time efficiency, degrading view transformation quality.
Approach
Lift image-segment prototypes into 3D voxel queries via feature aggregation and dispatch (prototype-aware view transformation), optimize prototype distinctiveness with pseudo-mask contrastive learning, and decode from augmented voxel views regularized for consistency.
Key results
- On Occ3D-nuScenes, ProtoOcc-Small (50x50x16 queries) reaches 37.80 mIoU, comparable to PanoOcc-Base at 38.11 with 200-level... (100x100x16) queries, with 60.53% less view-transformation compute.
- On SemanticKITTI, ProtoOcc improves VoxFormer-B mIoU from 12.35 to 13.57 and Symphonies-S mIoU from 13.64 to 14.50.
- Ablation on Occ3D: prototype mapping plus optimization plus multi-perspective decoding raises the Small-setting baseline from 35.78 to 37.80 mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Prototype Mapping only (ablation) | PanoOcc · Occ3D-nuScenes, Small | mIoU: 35.78 -> 35.80 | Helped |
| Add Prototype Optimization to Prototype Mapping (ablation) | PanoOcc · Occ3D-nuScenes, Small | mIoU: 35.80 -> 36.55 | Helped |
| Add Multi-perspective Occupancy Decoding only (ablation) | PanoOcc · Occ3D-nuScenes, Small | mIoU: 35.78 -> 37.25 | Helped |
| Add Prototype Mapping, Prototype Optimization, and Multi-perspective Occupancy Decoding (ablation) | PanoOcc · Occ3D-nuScenes, Small | mIoU: 35.78 -> 37.80 | Helped |
| Add consistency regularization to Random Dropout (ablation) | PanoOcc with Random Dropout · Occ3D-nuScenes, Small | mIoU: 36.99 -> 37.19 | Helped |
| Use 1440 prototypes instead of 350 (ablation) | ProtoOcc · Occ3D-nuScenes, Small | mIoU: 37.80 -> 36.90 | Hurt |
| Use 144 prototypes instead of 350 (ablation) | ProtoOcc · Occ3D-nuScenes, Small | mIoU: 37.80 -> 36.95 | Hurt |
| Use 91 prototypes instead of 350 (ablation) | ProtoOcc · Occ3D-nuScenes, Small | mIoU: 37.80 -> 37.07 | Hurt |
| Use Segment Anything rather than SEEDS for pseudo masks (ablation) | ProtoOcc with M=350 · Occ3D-nuScenes, Small | mIoU: 37.14 -> 37.29 | Helped |
| Add ProtoOcc to VoxFormer-S | VoxFormer-S · SemanticKITTI val | mIoU: 11.51 -> 12.39 (+0.88) | Helped |
| Add ProtoOcc to VoxFormer-B | VoxFormer-B · SemanticKITTI val | mIoU: 12.35 -> 13.57 (+1.22) | Helped |
| Add ProtoOcc to Symphonies-S | Symphonies-S · SemanticKITTI val | mIoU: 13.64 -> 14.50 (+0.86) | Helped |
Surprising findings
- Adding more voxel augmentations does not consistently improve results; combinations beyond two can hinder feature learning.
Limitations
- Gains on SemanticKITTI are small for some configurations (e.g. IoU drops for VoxFormer-B at smaller resolution not reported as improvement; Symphonies-B mIoU gain +0.45).
- Performance remains slightly below high-resolution baselines in some settings (e.g. 0.31 mIoU gap in Small-Base comparison).
Experimental setup
Datasets: Occ3D-nuScenes SemanticKITTI
Benchmarks: Occ3D-nuScenes SemanticKITTI
Metrics: mIoU IoU inference time FLOPs parameters
Baselines: BEVFormer PanoOcc CTF-Occ TPVFormer SurroundOcc OccFormer VoxFormer Symphonies MonoScene HASSC DFA3D
Builds on: PanoOcc BEVFormer VoxFormer Symphonies GRAND
Closest papers by meaning
- ProtoOcc: Accurate, Efficient 3D Occupancy Prediction Using Dual Branch Encoder-Prototype Query Decoder (AAAI 2025 · similarity 0.95)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.91)
- AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks (NeurIPS 2026 · similarity 0.90)
- LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-based 3D Semantic Occupancy Prediction (CVPR 2024 · similarity 0.89)
- ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction (AAAI 2025 · similarity 0.89)
- OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.