Research map / 3D vision / 3D generation
VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLScene, which distills LSeg vision-language semantic priors into image features and adds a geometric-semantic sparse awareness module to improve camera-based 3D semantic scene completion on SemanticKITTI and SSCBench-KITTI-360.
Problem
Camera-based 3D semantic scene completion suffers from geometric ambiguity due to occlusion and perspective distortion, and existing methods lack explicit semantic modeling between objects.
Approach
Distill high-level semantic priors from a vision-language model (LSeg) into the image semantic features via feature and logit distillation (VLGD), then refine voxel features with neighborhood geometry propagation (large/small 3D kernels) and sparse semantic interaction (Cylinder3D-style decomposed sparse kernels).
Key results
- Achieves 17.52 mIoU on the SemanticKITTI hidden test set, ahead of Symphonize (15.04) and BRGScene (15.36), using 47.4M parameters.
- Reaches 19.10 mIoU on SSCBench-KITTI-360 test set, versus 18.58 for Symphonies.
- On the SemanticKITTI validation set, VLGD raises mIoU from 15.43 to 17.83 when combined with the full architecture (ablation shows 2.4 mIoU over baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLScene instead of BRGScene | camera-based SSC · SemanticKITTI hidden test | mIoU: 15.36 -> 17.52 | Helped |
| Use VLScene instead of Symphonies | camera-based SSC · SSCBench-KITTI-360 test | mIoU: 18.58 -> 19.10 | Helped |
| Replace the baseline image encoder with RepViT (ablation) | VLScene architecture · SemanticKITTI val | mIoU: 15.43 -> 15.59 | Helped |
| Add VLGD to RepViT (ablation) | VLScene architecture · SemanticKITTI val | mIoU: 15.59 -> 16.58 | Helped |
| Add NGP to RepViT + VLGD (ablation) | VLScene architecture · SemanticKITTI val | mIoU: 16.58 -> 16.43 | Hurt |
| Add SSI to RepViT + VLGD (ablation) | VLScene architecture · SemanticKITTI val | mIoU: 16.58 -> 17.21 | Helped |
| Add both NGP and SSI to RepViT (ablation) | VLScene architecture · SemanticKITTI val | mIoU: 15.59 -> 16.66 | Helped |
| Add VLGD, NGP, and SSI to RepViT (ablation) | VLScene architecture · SemanticKITTI val | mIoU: 15.43 -> 17.83 | Helped |
| Use logits distillation only (ablation) | VLGD · SemanticKITTI val | mIoU: 15.43 -> 16.32 | Helped |
| Use feature distillation only (ablation) | VLGD · SemanticKITTI val | mIoU: 15.43 -> 16.54 | Helped |
| Combine feature and logits distillation (ablation) | VLGD · SemanticKITTI val | mIoU: 15.43 -> 17.83 | Helped |
| Replace GSSA with 3D Swin (ablation) | VLScene · SemanticKITTI val | mIoU: 17.83 -> 15.87 | Hurt |
Surprising findings
- Parameter count drops from 161.4M to 47.4M largely by swapping the baseline image encoder for RepViT, with a small mIoU change (15.43 to 15.59).
- Using only the NGP module (variant 3) gives higher IoU (44.58) than the variant with VLGD and SSI (44.03), while lower mIoU than VLGD+SSI.
Limitations
- Evaluated only in monocular/stereo settings; multi-camera SSC left to future work.
- Performance on some classes (e.g., other-ground, person, motorcyclist) remains low or inconsistent in tables.
Experimental setup
Datasets: SemanticKITTI SSCBench-KITTI-360
Benchmarks: SemanticKITTI SSCBench-KITTI-360
Metrics: mIoU IoU inference time parameters
Baselines: MonoScene TPVFormer OccFormer VoxFormer MonoOcc H2GFormer HASSC Symphonize BRGScene OctOcc
Builds on: BRGScene LSeg Cylinder3D SCPNet MonoScene
Closest papers by meaning
- Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance (NeurIPS 2025 · similarity 0.92)
- Bi-SSC: Geometric-Semantic Bidirectional Fusion for Camera-based 3D Semantic Scene Completion (CVPR 2024 · similarity 0.89)
- Memory-Augmented Re-Completion for 3D Semantic Scene Completion (AAAI 2025 · similarity 0.89)
- Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · similarity 0.89)
- Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion (CVPR 2025 · similarity 0.88)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.