Research map / 3D vision / Face and shape
MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs clothed, separated 3D avatars of multiple interacting people from a single monocular in-the-wild video using layered neural SDF representations, layer-wise volume rendering, and SAM-based progressive instance segmentation with confidence-guided pose/shape optimization.
Problem
Reconstructing complete, high-fidelity 3D shapes and appearance of multiple closely interacting people from a short monocular video, which requires pixel-level person disentanglement under occlusions and imperfect pose estimates.
Approach
A layered neural representation with per-person canonical SDF and texture fields composited via layer-wise occlusion-aware volume rendering, a closed-loop SAM prompting scheme driven by evolving human meshes for instance masks, and confidence-guided alternating optimization that freezes shape updates on unreliable frames.
Key results
- On Hi4D, the full method reaches V-IoU 0.816 and Chamfer 2.53 cm versus 0.783 and 3.02 cm for Vid2Avatar (per-person extension).
- Pose estimation on Hi4D improves MPJPE to 69.4 mm from 95.6 mm for TRACE and 85.7 mm for CLIFF.
- Novel view synthesis on Hi4D reaches PSNR 20.7 and LPIPS 0.0798, versus 19.6 and 0.1099 for Shuai et al.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add progressive SAM prompting to layer-wise volume rendering (ablation) | MultiPly · Hi4D | V-IoU: 0.773 -> 0.786 | Helped |
| Add progressive SAM prompting to layer-wise volume rendering (ablation) | MultiPly · Hi4D | C −ℓ2: 3.14 -> 2.68 | Hurt |
| Add confidence-guided optimization after progressive SAM (ablation) | MultiPly · Hi4D | V-IoU: 0.786 -> 0.816 | Helped |
| Add confidence-guided optimization after progressive SAM (ablation) | MultiPly · Hi4D | CD: 246.2 -> 218.4 | Helped |
| Use MultiPly instead of V2A for reconstruction | Hi4D human reconstruction · Hi4D | V-IoU: 0.783 -> 0.816 | Helped |
| Use MultiPly instead of V2A for reconstruction | MMM human reconstruction · MMM | V-IoU: 0.812 -> 0.826 | Helped |
| Use MultiPly instead of TRACE for pose estimation | Hi4D pose estimation · Hi4D | MPJPE: 95.6 -> 69.4 | Helped |
Surprising findings
- The initial SAM masks achieve segmentation comparable to SCHP but produce incomplete reconstructions, so progressive mesh-based prompting is needed.
Limitations
- Model complexity grows linearly with the number of persons, making it inefficient for crowds.
- Does not explicitly model hands.
- Per-person extension of Vid2Avatar is trained separately per subject in baselines.
Experimental setup
Datasets: Hi4D Monocular Multi-huMan (MMM)
Benchmarks: Hi4D Monocular Multi-huMan (MMM)
Metrics: V-IoU Chamfer distance P2S normal consistency PSNR SSIM LPIPS MPJPE MVE Contact Distance
Baselines: ECON Vid2Avatar Shuai et al. TRACE CLIFF SCHP
Builds on: Vid2Avatar NeRF++ SAM SMPL layered neural human rendering
Closest papers by meaning
- Vid2Avatar: 3D Avatar Reconstruction From Videos in the Wild via Self-Supervised Scene Decomposition (CVPR 2023 · similarity 0.90)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.88)
- Reconstructing People, Places, and Cameras (CVPR 2025 · similarity 0.88)
- Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses (CVPR 2024 · similarity 0.88)
- GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data (NeurIPS 2025 · similarity 0.87)
- MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.