Research map / 3D vision / 3D generation
COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COTR, a compact 3D occupancy transformer that combines explicit and implicit view transformation on a 1/16-size U-Net-downsampled occupancy feature with a semantic-aware group mask decoder using coarse-to-fine semantic grouping, improving TPVFormer, SurroundOcc, OccFormer and BEVDet4D on Occ3D-nuScenes.
Problem
Vision-based 3D occupancy prediction suffers from compressed views (TPV, BEV) losing geometry, while raw high-resolution occupancy is computationally redundant and sparse, and class imbalance hurts recognition of rare classes.
Approach
Build a compact occupancy feature via explicit view transformation (voxel pooling from depth-lifted features), downsample it in a U-Net, refine it with implicit view transformation (spatial cross-attention and self-attention), and decode with a mask-classification transformer whose queries are grouped coarse-to-fine by semantic granularity and sample count to balance supervision.
Key results
- On Occ3D-nuScenes, BEVDet4D + COTR reaches 44.5% mIoU with ResNet-50 at 256x704, outperforming VoxFormer and SurroundOcc by 3.8%.
- COTR improves TPVFormer, SurroundOcc, OccFormer and BEVDet4D mIoU by 5.1%, 4.7%, 3.8% and 5.2% respectively.
- Ablation on BEVDet4D: GOE, TD and CFSG together raise mIoU from 36.01% to 41.39% (+5.38) without long-term temporal information.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add COTR to BEVDet4D with ResNet-50 | BEVDet4D · Occ3D-nuScenes | mIoU: 39.3 -> 44.5 | Helped |
| Add COTR to BEVDet4D with SwinTransformer-B | BEVDet4D · Occ3D-nuScenes | mIoU: 42.5 -> 46.2 | Helped |
| Add geometry-aware occupancy encoder (ablation) | BEVDet4D baseline · Occ3D-nuScenes | IoU: 70.36 -> 70.89 | Helped |
| Add geometry-aware occupancy encoder (ablation) | BEVDet4D baseline · Occ3D-nuScenes | mIoU: 36.01 -> 37.97 | Helped |
| Add Transformer decoder (ablation) | BEVDet4D baseline · Occ3D-nuScenes | IoU: 70.36 -> 69.76 | Hurt |
| Add Transformer decoder (ablation) | BEVDet4D baseline · Occ3D-nuScenes | mIoU: 36.01 -> 38.43 | Helped |
| Add geometry-aware occupancy encoder and Transformer decoder (ablation) | BEVDet4D baseline · Occ3D-nuScenes | mIoU: 36.01 -> 40.22 | Helped |
| Add coarse-to-fine semantic grouping (ablation) | BEVDet4D with GOE and TD · Occ3D-nuScenes | mIoU: 40.22 -> 41.39 | Helped |
| Replicate ground-truth labels ten times instead of using one group (ablation) | BEVDet4D with GOE and TD · Occ3D-nuScenes | IoU: 71.74 -> 71.47 | Hurt |
| Replicate ground-truth labels ten times instead of using one group (ablation) | BEVDet4D with GOE and TD · Occ3D-nuScenes | mIoU: 40.22 -> 40.17 | Hurt |
| Add a {foreground, background, empty} group alongside ground-truth labels (ablation) | BEVDet4D with GOE and TD · Occ3D-nuScenes | mIoU: 40.22 -> 40.61 | Helped |
| Use multiple coarse-to-fine semantic groups instead of one ground-truth group (ablation) | BEVDet4D with GOE and TD and long-term temporal information · Occ3D-nuScenes | mIoU: 42.70 -> 44.45 | Helped |
Surprising findings
- Simply replicating ground-truth label groups ten times yields no mIoU gain, while adding a coarse {foreground, background, empty} group gives a significant boost.
- Replacing predicted semantics with ground-truth semantics in non-empty regions improves performance by about 95%, indicating semantic discriminability is the bottleneck.
Limitations
- Gains are reported on a single dataset (Occ3D-nuScenes).
- Compact representation downsampling inherently loses information for small objects, mitigated but not eliminated by U-Net.
Experimental setup
Datasets: Occ3D-nuScenes
Benchmarks: Occ3D-nuScenes
Metrics: IoU mIoU
Baselines: TPVFormer SurroundOcc OccFormer BEVDet4D FBOcc VoxFormer BEVFormer MonoScene
Builds on: BEVStereo MaskFormer BEVDet4D TPVFormer SurroundOcc
Closest papers by meaning
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.91)
- SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction (CVPR 2024 · similarity 0.89)
- Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles (CVPR 2024 · similarity 0.89)
- SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction (CVPR 2025 · similarity 0.89)
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.89)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.