Research map / 3D vision / Depth and stereo
PMNI: Pose-free Multi-view Normal Integration for Reflective and Textureless Surface Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMNI, a pose-free multi-view surface normal integration method that jointly optimizes a hash-encoded neural SDF and camera poses from multi-view normal maps to reconstruct reflective and textureless surfaces.
Problem
Multi-view 3D reconstruction of reflective and textureless surfaces fails when camera poses must be calibrated or estimated from unreliable cross-view RGB features.
Approach
Replace RGB inputs with photometric-stereo surface normal maps, use per-view depth from normal integration (BiNI) as an anchor for an SDF, and enforce multi-view normal consistency at projected ray-surface correspondences to jointly recover shape and camera poses without calibration.
Key results
- On DiLiGenT-MV, PMNI achieves average CD 0.153 versus 0.142 for calibrated-pose SuperNormal, with average F1 0.990 vs 0.993.
- On the RT3D dataset, PMNI reduces average rotation error to 0.354 degrees and translation error to 0.022, versus 2.920 and 0.304 for DUSt3R (best baseline).
- With noisy poses, SuperNormal's CD rises to 0.775 average while PMNI stays at 0.153.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Lnormal (ablation) | PMNI · POT2 (DiLiGenT-MV) | CD: 0.691 -> 0.115 | Helped |
| add Lni (ablation) | PMNI · POT2 (DiLiGenT-MV) | CD: 0.163 -> 0.115 | Helped |
| add Lc (ablation) | PMNI · POT2 (DiLiGenT-MV) | CD: 0.126 -> 0.115 | Helped |
| jointly optimize camera poses and surface shape rather than using noisy calibrated poses | SuperNormal reconstruction · DiLiGenT-MV | CD: 0.775 -> 0.153 | Helped |
| use PMNI instead of DUSt3R | RT3D · RT3D (average) | RPEr (°): 2.920 -> 0.354 | Helped |
| use PMNI instead of SPARF | RT3D · RT3D (average) | Relative Depth Error: 0.067 -> 0.014 | Helped |
Surprising findings
- Pose-free PMNI nearly matches calibrated-pose SuperNormal in shape accuracy despite not using calibrated poses.
Limitations
- Relies on per-view surface normal maps from photometric stereo, which require multi-light capture.
- Performance degrades with noisy input normals (e.g., StableNormal with 23.6 degree MAE gives CD 0.602).
- Requires foreground masks as input.
Experimental setup
Datasets: DiLiGenT-MV RT3D
Benchmarks: DiLiGenT-MV RT3D
Metrics: Chamfer distance F1-score RPEr RPEt relative depth error
Baselines: SuperNormal DUSt3R SPARF Nope-NeRF CF-3DGS
Builds on: NeuS BiNI SuperNormal BARF Nope-NeRF
Closest papers by meaning
- SuperNormal: Neural Surface Reconstruction via Multi-View Normal Integration (CVPR 2024 · similarity 0.88)
- SC-NeuS: Consistent Neural Surface Reconstruction from Sparse and Noisy Views (AAAI 2024 · similarity 0.87)
- Differentiable Neural Surface Refinement for Modeling Transparent Objects (CVPR 2024 · similarity 0.87)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.86)
- NeuSurf: On-Surface Priors for Neural Surface Reconstruction from Sparse Input Views (AAAI 2024 · similarity 0.86)
- NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.