Research map / 3D vision / Face and shape
MorpheuS: Neural Dynamic 360deg Surface Reconstruction from Monocular RGB-D Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MorpheuS, a neural implicit framework that reconstructs dynamic 360-degree object surfaces from casual monocular RGB-D video by distilling a Zero-1-to-3 diffusion prior via SDS into a hyper-dimensional canonical SDF/color field with a deformation field.
Problem
Dynamic surface reconstruction from casually captured monocular RGB-D video leaves large unobserved regions that neural representations fail to complete realistically, especially for 360-degree views of deformable objects.
Approach
Combines a hyper-dimensional canonical field and deformation field (HyperNeRF/NDR style) with view-conditioned diffusion prior SDS, using temporal keyframe-conditioned, view-angle-weighted SDS and a canonical-space regularization applied to sampled points instead of observation-space points.
Key results
- On 9 real-world scenes, average accuracy improves from 1.09 cm (NDR completion) to 0.78 cm completion and CLIP similarity rises from 79.75 to 86.77 versus NDR.
- On synthetic AMA and BANMo scenes, average CLIP similarity is 91.31 vs 85.65 for NDR, with average accuracy 2.57 cm vs 6.73 cm.
- Ablation: removing the diffusion prior lowers CLIP similarity from 86.77 to 82.68 and removing depth raises accuracy error from 0.88 cm to 3.24 cm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MorpheuS instead of NDR | Dynamic surface reconstruction · 9 real-world scenes (Avg.) | Acc. [cm] ↓: 1.92 -> 0.88 | Helped |
| Use MorpheuS instead of NDR | Dynamic surface reconstruction · 9 real-world scenes (Avg.) | CLIP sim. ↑: 79.75 -> 86.77 | Helped |
| Use MorpheuS instead of NDR | Dynamic surface reconstruction · 4 synthetic scenes (Avg.) | Acc. [cm] ↓: 6.73 -> 2.57 | Helped |
| Use MorpheuS instead of NDR | Dynamic surface reconstruction · 4 synthetic scenes (Avg.) | CLIP sim. ↑: 85.65 -> 91.31 | Helped |
| Add a diffusion prior (ablation) | MorpheuS · 9 real-world scenes | Acc. [cm] ↓: 0.99 -> 0.88 | Helped |
| Add a diffusion prior (ablation) | MorpheuS · 9 real-world scenes | CLIP sim. ↑: 82.68 -> 86.77 | Helped |
| Add temporal keyframe conditioning instead of conditioning on the first frame (ablation) | MorpheuS diffusion prior · 9 real-world scenes | Comp. [cm] ↓: 0.95 -> 0.78 | Helped |
| Add temporal keyframe conditioning instead of conditioning on the first frame (ablation) | MorpheuS diffusion prior · 9 real-world scenes | CLIP sim. ↑: 86.01 -> 86.77 | Helped |
| Add view-angle weighting to the SDS loss (ablation) | MorpheuS · 9 real-world scenes | Comp. [cm] ↓: 0.86 -> 0.78 | Helped |
| Add depth measurement to RGB supervision (ablation) | MorpheuS · 9 real-world scenes | Acc. [cm] ↓: 3.24 -> 0.88 | Helped |
| Add depth measurement to RGB supervision (ablation) | MorpheuS · 9 real-world scenes | Comp. [cm] ↓: 4.90 -> 0.78 | Helped |
| Use a canonical representation instead of a unified 4D space-time field (ablation) | MorpheuS · 9 real-world scenes | Comp. [cm] ↓: 1.05 -> 0.78 | Helped |
Surprising findings
- Without the diffusion prior, reconstruction accuracy in observed regions is worse (0.99 vs 0.88 cm), indicating the diffusion prior also regularizes observed geometry.
Limitations
- Fails on incomplete views, motion blur, and articulated poses.
- Cannot reconstruct complex motion in self-occluded regions due to lack of motion priors.
- Diffusion prior conditioned only on RGB frames, not RGB-D.
Experimental setup
Datasets: KillingFusion DeepDeform iPhone dataset AMA BANMo
Benchmarks: KillingFusion DeepDeform iPhone AMA BANMo
Metrics: accuracy (cm) completion (cm) CLIP similarity
Baselines: NDR Magic123
Builds on: NDR HyperNeRF DreamFusion Zero-1-to-3 VolSDF
Closest papers by meaning
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (CVPR 2024 · similarity 0.88)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.87)
- Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels (NeurIPS 2024 · similarity 0.87)
- Differentiable Neural Surface Refinement for Modeling Transparent Objects (CVPR 2024 · similarity 0.87)
- MonoNPHM: Dynamic Head Reconstruction from Monocular Videos (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.