Research map / 3D vision / 3D generation

Semi-supervised 3D Semantic Scene Completion with 2D Vision Foundation Model Guidance

AAAI 2025 · 3D vision › 3D generation · approach: Semantic scene completion · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VFG-SSC, a semi-supervised vision-based 3D semantic scene completion framework that uses 3D clues from 2D depth and segmentation foundation models, fused via a local-attention enhancement module, to generate pseudo-labels for unlabeled images.

Problem

Vision-based 3D semantic scene completion normally needs dense LiDAR-derived voxel labels, which are expensive; the paper targets semi-supervised SSC where only a small fraction of images have 3D occupancy labels.

Approach

Accumulate depth and semantic maps from temporal image sequences, unproject them into semantic 3D clues with geometric/semantic filtering, and fuse these clues with SSC backbone features via dilated neighborhood attention to produce better pseudo-labels for self-training; the module plugs into different SSC backbones.

Key results

• On SemanticKITTI validation with 10% labels, VFG-SSC on VoxFormer reaches 12.19 mIoU / 41.57 IoU versus 14.40 mIoU / 43.50 IoU fully supervised.
• On NYUv2 test with 10% labels, VFG-SSC on OccDepth reaches 23.92 mIoU vs 29.03 fully supervised, and on MonoScene reaches 22.20 vs 26.94 mIoU.
• With 1% labels on SemanticKITTI using OccFormer, VFG-SSC improves mIoU from 6.59 (supervised-only) to 9.40.

Measured effects reported in the paper

Surprising findings

• Simply concatenating or fusing VFM 3D clues into an SSC network gives little gain over the baseline without the proposed enhancement module.
• Adding temporal 3D clues without regularization decreases mIoU (-0.62), and Mean-Teacher gives better IoU than VFG-SSC on VoxFormer at 1% labels.

Limitations

• Relies on 2D vision foundation models, which may fail at night, in fog, or rain.
• Large domain gaps between training and test data may hurt performance since only a small amount of labeled data is used.
• Gap to fully supervised remains about 15% on SemanticKITTI.

Experimental setup

Datasets: SemanticKITTI NYUv2 KITTI

Benchmarks: SemanticKITTI NYUv2

Metrics: mIoU IoU

Baselines: Supervised-only Self-Training Mean-Teacher SelfOcc SceneRF 3D clues as pseudo-labels

Builds on: Self-Training Mean Teacher Dilated Neighborhood Attention VoxFormer OccFormer

Closest papers by meaning

• H2GFormer: Horizontal-to-Global Voxel Transformer for 3D Semantic Scene Completion (AAAI 2024 · similarity 0.90)
• Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving (ICLR 2025 · similarity 0.88)
• VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion (AAAI 2025 · similarity 0.88)
• LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-based 3D Semantic Occupancy Prediction (CVPR 2024 · similarity 0.88)
• VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction (CVPR 2025 · similarity 0.88)
• LOMA: Language-assisted Semantic Occupancy Network via Triplane Mamba (AAAI 2025 · similarity 0.87)