Research map / 3D vision / 3D generation
Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlowScene, a temporal semantic scene completion method that uses GMFlow optical flow and forward-backward occlusion masks to align and aggregate historical RGB features and refine 3D voxel features.
Problem
Camera-based 3D semantic scene completion for autonomous driving struggles to exploit temporal context: existing methods use only current-frame cues or naively stack multi-frame features, ignoring motion dynamics and temporal consistency.
Approach
Use pre-trained optical flow (GMFlow, frozen) with forward-backward consistency occlusion masks to warp historical 2D features to the current frame (flow-guided temporal aggregation), then inject non-occluded aggregated features and occlusion masks into 3D voxel space via LSS for occlusion-guided voxel refinement.
Key results
- Achieves 17.70 mIoU on the SemanticKITTI hidden test set, above HTCL (17.09) and CGFormer (16.63).
- Achieves 20.81 mIoU on SSCBench-KITTI-360, above CGFormer (20.05).
- On the SemanticKITTI validation set, the flow-guided variant reaches 18.13 mIoU with 52.4M parameters; ablation shows the full model improves over the stacked baseline by 2.24 mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add flow-guided warping to stacked temporal features (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 15.89 -> 16.21 | Helped |
| add occlusion detection and occlusion cross-attention (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 16.21 -> 16.43 | Helped |
| add temporal aggregation (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 16.67 -> 17.23 | Helped |
| add occlusion cross-attention (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 16.67 -> 17.08 | Helped |
| add the voxel occlusion mask to convolution-based voxel fusion (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 17.63 -> 18.13 | Helped |
| replace stacked temporal alignment with optical-flow-guided alignment (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 15.89 -> 18.13 | Helped |
| increase historical frames from two to three (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 18.13 -> 18.30 | Helped |
| increase historical frames from two to four or five (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 18.13 -> 17.68; 18.13 -> 17.55 | Hurt |
| replace GMFlow with alternative optical-flow networks (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 18.13 -> 17.13; 18.13 -> 17.56; 18.13 -> 17.74 | Hurt |
| replace RepViT-M2.3 with ResNet50 or EfficientNetB7 (ablation) | FlowScene on SemanticKITTI validation · SemanticKITTI validation | mIoU: 18.13 -> 16.98; 18.13 -> 17.63 | Hurt |
| use FlowScene instead of HTCL | SemanticKITTI hidden test · SemanticKITTI hidden test | mIoU: 17.09 -> 17.70 | Helped |
| use FlowScene instead of CGFormer | SSCBench-KITTI-360 test · SSCBench-KITTI-360 test | mIoU: 20.05 -> 20.81 | Helped |
Surprising findings
- Reproducing prior SOTA methods with the RepViT encoder changed their parameter counts and sometimes slightly lowered mIoU (e.g., HTCL 17.13 to 16.86).
- Adding more historical frames beyond two (t-3 to t-5) reduced IoU and mIoU, which the authors attribute to degraded optical flow over longer intervals.
Limitations
- Inference time needs improvement, according to the authors.
- Depends on a pretrained optical flow model, which may degrade in adverse visual conditions.
- Performance drops with more temporal frames because flow quality degrades over longer intervals.
Experimental setup
Datasets: SemanticKITTI SSCBench-KITTI-360 ImageNet
Benchmarks: SemanticKITTI SSCBench-KITTI-360
Metrics: IoU mIoU
Baselines: MonoScene TPVFormer OccFormer VoxFormer-T Symphonize BRGScene CGFormer H2GFormer-T HASSC-T SGN HTCL VLScene
Builds on: FGFA LSS MonoScene BEVDepth GMFlow
Closest papers by meaning
- VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion (AAAI 2025 · similarity 0.92)
- Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion (CVPR 2025 · similarity 0.89)
- Memory-Augmented Re-Completion for 3D Semantic Scene Completion (AAAI 2025 · similarity 0.89)
- Context and Geometry Aware Voxel Transformer for Semantic Scene Completion (NeurIPS 2024 · similarity 0.89)
- Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos (ICLR 2024 · similarity 0.88)
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.