Research map / Vision-language and multimodal / Cross-modal learning
Unleashing Network Potentials for Semantic Scene Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMMNet, an RGB-D semantic scene completion network with cross-modal feature modulation for gradient interdependence between RGB and TSDF encoders, plus an adversarial discriminator trained on geometrically and semantically perturbed ground truths to reduce overfitting.
Problem
Multi-modal semantic scene completion from single-view RGB-D suffers from under-utilized single-modality encoder representations after joint training and overfitting on small datasets like NYU.
Approach
Replace additive RGB-TSDF fusion with a cross-modal modulation (scale and bias from TSDF features applied to RGB features) so gradients flow interdependently across modalities, and add a minimax adversarial discriminator whose fake samples are GT with erased voxels (geometry) or shuffled class labels (semantics).
Key results
- AMMNet reaches 48.5% SSC-mIoU on NYU test, 3.5% above CVSformer, the previous best; the DeepLabv3 variant reaches 56.1%.
- On NYUCAD, AMMNet-DeepLabv3 reaches 67.2% SSC-mIoU, the top score in the comparison.
- Ablation on NYU: the baseline at 46.1% SSC-mIoU and 71.6% SC-IoU rises to 48.5% and 75.6% with both components.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AMMNet with DeepLabv3 backbone | SSC models on NYU · NYU test | SSC-mIoU: 3.5% | Helped |
| add AMMNet with DeepLabv3 backbone | SSC models on NYUCAD · NYUCAD test | SSC-mIoU: 3.3% | Helped |
| add cross-modal modulation (ablation) | baseline SSC model · NYU test | SSC-mIoU: 2.4% | Helped |
| add adversarial training (ablation) | baseline SSC model · NYU test | SSC-mIoU: 1.6% | Helped |
| add cross-modal modulation (ablation) | adversarially trained SSC model · NYU test | SSC-mIoU: 0.8% | Helped |
| add M2nd and M3rd modulation (ablation) | AMMNet without adversarial training · NYU test | SSC-mIoU: 47.1% -> 47.4% | Helped |
| add M2nd and M3rd modulation (ablation) | AMMNet with adversarial training · NYU test | SSC-mIoU: 47.8% -> 48.5% | Helped |
| retain forward feature calibration but detach inter-dependent modulation gradients (ablation) | AMMNet⋆ · NYU test | SSC-mIoU: 47.1% -> 46.9% | Hurt |
| replace modulation with addition while retaining inter-dependent gradient updating (ablation) | AMMNet⋆ · NYU test | SSC-mIoU: 47.1% -> 46.5% | Hurt |
| add geometry-perturbed fake samples (ablation) | AMMNet† with discriminator · NYU test | SSC-mIoU: 46.7% -> 46.6% | No clear effect |
| add semantic perturbation (ablation) | AMMNet† with geometry perturbation · NYU test | SSC-mIoU: 46.6% -> 47.1% | Helped |
Surprising findings
- Multi-modal encoders learned under joint training yield weaker single-modal representations than uni-modal training (e.g., 0.37% and 0.51% SSC-mIoU drops)
- The baseline model overfits: training mIoU rises 22.8% in later epochs while validation mIoU drops 2.8%
Limitations
- Evaluated only on NYU and NYUCAD indoor datasets with single-view RGB-D input
- Gains from modulation beyond adversarial training are modest in some ablations (e.g., +0.8% SSC-mIoU over strong baseline)
- Hyperparameters such as beta and perturbation probabilities tuned on a small random validation split
Experimental setup
Datasets: NYU NYUCAD
Benchmarks: NYU NYUCAD
Metrics: SSC-mIoU SC-IoU precision recall
Baselines: SSCNet CCPNet DDRNet AIC-Net 3D-Sketch FFNet CleanerS PCANet SISNet CVSformer
Builds on: CleanerS 3D-Sketch DDRNet SegFormer
Closest papers by meaning
- Memory-Augmented Re-Completion for 3D Semantic Scene Completion (AAAI 2025 · similarity 0.90)
- Context and Geometry Aware Voxel Transformer for Semantic Scene Completion (NeurIPS 2024 · similarity 0.89)
- Voxel Proposal Network via Multi-Frame Knowledge Distillation for Semantic Scene Completion (NeurIPS 2024 · similarity 0.87)
- Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance (NeurIPS 2025 · similarity 0.87)
- Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation (CVPR 2024 · similarity 0.87)
- H2GFormer: Horizontal-to-Global Voxel Transformer for 3D Semantic Scene Completion (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.