Research map / 3D vision / Depth and stereo
Shading Meets Motion: Self-supervised Indoor 3D Reconstruction Via Simultaneous Shape-from-Shading and Structure-from-Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shading-SfM-Net jointly trains a self-supervised shape-from-shading network (inverse rendering) and a structure-from-motion network with keypoint, layout, and geometric consistency losses to improve depth and 3D reconstruction on low-texture indoor scenes.
Problem
Self-supervised monocular depth and structure-from-motion fail on low-texture, repeated-pattern indoor regions (walls, floors) where photometric constraints provide little signal, degrading depth and camera pose estimation.
Approach
Couple an inverse-rendering shape-from-shading branch (albedo, normals, lighting) with a monocular SfM branch through surface shape consistency (superpixel curved-surface fitting on albedo segments), cross-normal consistency, and 3D scene registration losses, plus BRISK-keypoint photometric loss and line-based room layout consistency.
Key results
- On NYUv2, full model reaches AbsRel 0.129 and delta1 0.828, the best among listed unsupervised methods (RA-Depth 0.152 AbsRel).
- On ScanNet, full model reaches AbsRel 0.151 and delta1 0.798, outperforming unsupervised baselines such as RA-Depth (0.169) and P2Net (0.181).
- Ablation on NYUv2 shows AbsRel improving from 0.178 (no SfS-Net) to 0.129 with all components.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SfS-Net (ablation) | SfM-Net · NYUv2 | AbsRel: 0.178 -> 0.161 | Helped |
| add surface shape consistency (SSC) (ablation) | SfS-Net · NYUv2 | AbsRel: 0.161 -> 0.155 | Helped |
| add scene registration consistency (SRC) (ablation) | SfS-Net + SSC · NYUv2 | AbsRel: 0.155 -> 0.149 | Helped |
| add keypoint-based consistency (KPC) (ablation) | SfS-Net + SSC + SRC · NYUv2 | AbsRel: 0.149 -> 0.138 | Helped |
| add layout prediction consistency (LPC) (ablation) | SfS-Net + SSC + SRC + KPC · NYUv2 | AbsRel: 0.138 -> 0.129 | Helped |
Surprising findings
- Several prior unsupervised methods (e.g., GeoNet) collapsed during training on NYUv2, per the reported table.
Limitations
- Gains over supervised methods are only comparable rather than superior on depth metrics.
- Evaluated mainly on NYUv2 and ScanNet; outdoor FinnForest evaluated only qualitatively.
- Relies on external line detectors (LayoutNet/RoomNet) for layout consistency.
Experimental setup
Datasets: NYUv2 ScanNet FinnForest
Benchmarks: NYUv2 ScanNet
Metrics: AbsRel RMS Log10 delta1 delta2 delta3
Baselines: Monodepth2 MovingIndoor TrianFlow PackNet-SfM P2Net HR-Depth Lite-Mono RA-Depth SC-SfMLearner DORN
Builds on: SfMLearner Monodepth2 P2Net SfSNet MovingIndoor
Closest papers by meaning
- Behind the Veil: Enhanced Indoor 3D Scene Reconstruction with Occluded Surfaces Completion (CVPR 2024 · similarity 0.85)
- Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping (NeurIPS 2025 · similarity 0.85)
- IndoorGS: Geometric Cues Guided Gaussian Splatting for Indoor Scene Reconstruction (CVPR 2025 · similarity 0.84)
- MvFFN: Multi-view Floor-Plan Feed-Forward Network for Unposed Wide-Baseline Panorama Layout Reconstruction (NeurIPS 2026 · similarity 0.84)
- ND-SDF: Learning Normal Deflection Fields for High-Fidelity Indoor Reconstruction (ICLR 2025 · similarity 0.84)
- MP-SfM: Monocular Surface Priors for Robust Structure-from-Motion (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.