Research map / 3D vision / 3D generation
Symphonize 3D Semantic Scene Completion with Contextual Instance Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Symphonies, a camera-only 3D semantic scene completion method that uses contextual instance queries and Serial Instance-Propagated Attentions to lift image features into voxel volumes, reaching SOTA mIoU on SemanticKITTI and SSCBench-KITTI-360.
Problem
Vision-based 3D semantic scene completion relies on voxel-wise feature aggregation and pixel-voxel projection, which neglects instance semantics and scene context and leads to geometric ambiguities from occlusion and perspective errors.
Approach
Uses instance queries derived from images as intermediaries in a decoder that iteratively propagates image features to voxels via instance-image, instance-scene, scene self, and instance self attentions, combined with a depth-rectified voxel proposal layer initializing voxels on the estimated implicit surface.
Key results
- Achieves 15.04 mIoU on SemanticKITTI test, a 2.72 mIoU gain over prior camera-based methods.
- Achieves 18.58 mIoU on SSCBench-KITTI-360 test, a 4.77 mIoU gain over prior camera-based methods.
- Ablation: the Symphonies Decoder raises validation IoU from 36.54 to 41.92 and mIoU from 13.34 to 14.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Symphonies instead of OccFormer | Camera-based SSC · SemanticKITTI test | mIoU: 12.32 -> 15.04 | Helped |
| Use Symphonies instead of VoxFormer | Camera-based SSC · SemanticKITTI test | IoU: 42.95 -> 42.19 | Hurt |
| Use Symphonies instead of OccFormer | Camera-based SSC · SSCBench-KITTI-360 test | mIoU: 13.81 -> 18.58 | Helped |
| Add pre-trained weights to the Image Encoder (ablation) | Symphonies baseline · SemanticKITTI val | mIoU: 10.44 -> 12.59 (+2.15) | Helped |
| Add the Depth-Rectified Voxel Proposal Layer (ablation) | Symphonies with pre-trained encoder · SemanticKITTI val | mIoU: 12.59 -> 13.34 (+0.75) | Helped |
| Add the Symphonies Decoder (ablation) | Symphonies with Voxel Proposal Layer · SemanticKITTI val | mIoU: 13.34 -> 14.89 (+1.55) | Helped |
| Add the Symphonies Decoder (ablation) | Symphonies with Voxel Proposal Layer · SemanticKITTI val | IoU: 36.54 -> 41.92 (+5.38) | Helped |
| Add Instance Self-Attention to the decoder (ablation) | Symphonies Decoder · SemanticKITTI val | mIoU: 14.37 -> 14.63 | Helped |
| Add Scene Self-Attention to the decoder (ablation) | Symphonies Decoder · SemanticKITTI val | mIoU: 14.73 -> 14.89 | Helped |
| Use stereo depth estimation instead of FLoSP projection (ablation) | Voxel Proposal Layer · SemanticKITTI val | mIoU: 11.96 -> 14.89 | Helped |
| Use stereo depth estimation instead of monocular depth estimation (ablation) | Voxel Proposal Layer · SemanticKITTI val | mIoU: 12.20 -> 14.89 | Helped |
| Combine pre-trained Image Encoder weights with instance queries (ablation) | Symphonies · SemanticKITTI val | mIoU: 13.53 -> 14.89 | Helped |
Surprising findings
- Instance-image cross-attention brings less improvement than instance-scene cross-attention, suggesting image features are already captured by the voxel proposal layer.
Limitations
- Trails VoxFormer in IoU on SemanticKITTI, attributed to its two-stage design rather than this method
- Relies on a pre-trained stereo depth estimator (MobileStereoNet) for voxel proposals
- Sparse instance deformable attentions cause training fluctuations unless scene self-attention is included
Experimental setup
Datasets: SemanticKITTI SSCBench-KITTI-360
Benchmarks: SemanticKITTI SSCBench-KITTI-360
Metrics: IoU mIoU
Baselines: MonoScene TPVFormer VoxFormer OccFormer LMSCNet AICNet JS3C-Net SSCNet
Builds on: MonoScene VoxFormer MaskDINO Deformable DETR MobileStereoNet
Closest papers by meaning
- VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion (AAAI 2025 · similarity 0.87)
- Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion (CVPR 2025 · similarity 0.87)
- Context and Geometry Aware Voxel Transformer for Semantic Scene Completion (NeurIPS 2024 · similarity 0.87)
- Memory-Augmented Re-Completion for 3D Semantic Scene Completion (AAAI 2025 · similarity 0.87)
- Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance (NeurIPS 2025 · similarity 0.87)
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.