Research map / 3D vision / Gaussian splatting
Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs a static background and multiple dynamic humans from a monocular video as composable 3D Gaussian Splatting, using SDS with a personalized (textual-inversion) Stable Diffusion and OpenPose ControlNet to hallucinate unseen body parts.
Problem
Reconstructing dynamic 3D scenes with multiple humans from monocular in-the-wild video, where humans are occluded, cropped, or sparsely observed so large body parts are never seen.
Approach
Represent background and each human as 3D Gaussians (humans in SMPL canonical space with LBS deformation) and apply score distillation sampling from a person-specific, textual-inversion-personalized latent diffusion model with OpenPose ControlNet on random virtual views and poses to complete unseen regions consistently with observed appearance.
Key results
- On Panoptic, the method reaches 23.60 PSNR versus 19.59 for HumanNeRF and 15.79 for Shuai et al.
- On Hi4D pair01-hug, PSNR is 25.14 versus 21.40 for HumanNeRF and 19.72 for Shuai et al.
- Rendering runs at 361 FPS at 512x512 versus 18 FPS for InstantAvatar, with 1496MiB VRAM versus 4972MiB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed 3D-GS reconstruction with diffusion guidance | monocular multi-person reconstruction · Panoptic | PSNR: 15.79 -> 23.60 | Helped |
| use the proposed 3D-GS reconstruction with diffusion guidance | monocular multi-person reconstruction · Hi4D pair00-dance | PSNR: 20.78 -> 23.76 | Helped |
| use the proposed 3D-GS reconstruction with diffusion guidance | monocular multi-person reconstruction · Hi4D pair01-hug | PSNR: 21.40 -> 25.14 | Helped |
| use the proposed method when observations are sufficient | single-person reconstruction · ZJU-Mocap 6 subjects | PSNR: 30.23 -> 30.13 | No clear effect |
| add SDS loss (ablation) | Ours on ZJU-Mocap 377 subject with lower-body occlusion · ZJU-Mocap 377 subject, lower-body occluded | PSNR: 24.51 -> 26.03 | Helped |
| add textual inversion (ablation) | Ours on ZJU-Mocap 377 subject with lower-body occlusion · ZJU-Mocap 377 subject, lower-body occluded | PSNR: 23.43 -> 26.03 | Helped |
| add reconstruction loss (ablation) | Ours on ZJU-Mocap 377 subject with lower-body occlusion · ZJU-Mocap 377 subject, lower-body occluded | PSNR: 23.43 -> 26.03 | Helped |
Surprising findings
- On ZJU-Mocap with full observations, HumanNeRF still has better LPIPS (3.36 vs 4.50), so the method is only comparable there.
Limitations
- Requires SMPL fitting to be provided as input
- Only humans are modeled as dynamic objects, ignoring animals and other moving objects
- Synthesized unseen parts still show visible artifacts
Experimental setup
Datasets: Panoptic Dataset Hi4D ZJU-Mocap
Benchmarks: Panoptic Dataset Hi4D ZJU-Mocap
Metrics: PSNR SSIM LPIPS rendering FPS VRAM
Baselines: HumanNeRF InstantAvatar Shuai et al. (ST-NeRF/NeuralBody compositional)
Builds on: 3D Gaussian Splatting DreamFusion SDS CustomDiffusion HumanNeRF LBS skinning
Closest papers by meaning
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.91)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.90)
- HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors (NeurIPS 2024 · similarity 0.90)
- Reconstructing People, Places, and Cameras (CVPR 2025 · similarity 0.89)
- RoGSplat: Learning Robust Generalizable Human Gaussian Splatting from Sparse Multi-View Images (CVPR 2025 · similarity 0.89)
- Generative Sparse-View Gaussian Splatting (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.