Research map / 3D vision / 3D generation
Context and Geometry Aware Voxel Transformer for Semantic Scene Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CGFormer, a vision-based semantic scene completion network with context-aware voxel queries, 3D deformable cross-attention in 3D pixel space, a stereo-refined depth net, and a voxel+TPV local-global encoder.
Problem
Vision-based semantic scene completion suffers from context-independent shared queries in sparse-to-dense transformers, undirected cross-attention aggregation, and depth ambiguity when 3D points project to similar 2D image locations.
Approach
Generate context-dependent voxel queries from each image's context and depth features, extend deformable cross-attention from 2D to 3D pixel space via outer product of context and depth probability so points with the same image coordinates are separated by depth, and fuse voxel and TPV branches with dynamic weighting.
Key results
- CGFormer reaches 16.87 mIoU and 45.99 IoU on the SemanticKITTI validation set, and 16.63 mIoU / 44.41 IoU on the hidden test set.
- On SSCBench-KITTI-360 test, CGFormer achieves 20.05 mIoU and 48.07 IoU, surpassing all camera-based baselines by 3.95 IoU and 1.67 mIoU.
- Replacing 3D deformable cross-attention gives +1.63 mIoU and +2.15 IoU over the baseline (ablation a vs. baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3D deformable cross-attention (ablation) | CGFormer baseline · SemanticKITTI validation | mIoU: 12.71 -> 14.34 | Helped |
| add context-aware query generation (ablation) | CGFormer with 3D-DCA · SemanticKITTI validation | mIoU: 14.34 -> 15.60 | Helped |
| add local voxel and TPV branches with dynamic fusion (ablation) | CGFormer with CGVT · SemanticKITTI validation | mIoU: 15.60 -> 16.87 | Helped |
| use all three TPV planes with dynamic fusion instead of one plane (ablation) | CGFormer LGE · SemanticKITTI validation | mIoU: 16.38; 16.74; 16.49 -> 16.87 | Mixed |
| replace CAQG with more attention layers (ablation) | CGFormer · SemanticKITTI validation | mIoU: 14.34 -> 14.43 | Helped |
| replace CAQG with FLoSP (ablation) | CGFormer · SemanticKITTI validation | mIoU: 14.34 -> 14.66 | Helped |
| use voxel pooling as the CAQG (ablation) | CGFormer · SemanticKITTI validation | mIoU: 14.34 -> 15.60 | Helped |
| add the stereo feature to depth refinement (ablation) | CGFormer depth net · SemanticKITTI validation | mIoU: 15.08 -> 16.87 | Helped |
| add neighborhood attention to depth refinement (ablation) | CGFormer depth net · SemanticKITTI validation | mIoU: 16.26 -> 16.87 | Helped |
| replace the depth refinement module with StereoScene dense correlation (ablation) | CGFormer depth net · SemanticKITTI validation | mIoU: 16.49 -> 16.87 | Hurt |
| replace EfficientNetB7 with ResNet50 while retaining Swin blocks (ablation) | CGFormer · SemanticKITTI validation | mIoU: 16.87 -> 16.79 | No clear effect |
| replace EfficientNetB7 with ResNet50 and Swin blocks with ResBlocks (ablation) | CGFormer · SemanticKITTI validation | mIoU: 16.87 -> 16.85 | No clear effect |
Surprising findings
- Adding more cross-attention and self-attention layers raises IoU but barely changes mIoU while greatly increasing memory, so the context-aware query generator is the more efficient choice.
Limitations
- Accuracy on many categories such as person, bicyclist and other vehicle remains low.
- Depth estimation is designed for single-view or stereo settings and does not extend to multi-view scenarios.
Experimental setup
Datasets: SemanticKITTI SSCBench-KITTI-360
Benchmarks: SemanticKITTI SSCBench-KITTI-360
Metrics: IoU mIoU training memory inference time parameters
Baselines: MonoScene TPVFormer SurroundOcc OccFormer IAMSSC VoxFormer DepthSSC Symphonize HASSC StereoScene H2GFormer MonoOcc
Builds on: VoxFormer MonoScene TPVFormer BEVDepth StereoScene
Closest papers by meaning
- H2GFormer: Horizontal-to-Global Voxel Transformer for 3D Semantic Scene Completion (AAAI 2024 · similarity 0.93)
- Memory-Augmented Re-Completion for 3D Semantic Scene Completion (AAAI 2025 · similarity 0.90)
- Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion (CVPR 2025 · similarity 0.89)
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.89)
- Voxel Proposal Network via Multi-Frame Knowledge Distillation for Semantic Scene Completion (NeurIPS 2024 · similarity 0.89)
- Unleashing Network Potentials for Semantic Scene Completion (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.