Research map / 3D vision / Gaussian splatting
World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Conditions a Wan-2.1-14B video DiT on dense 4D buffers (RGB, opacity, depth, normals, 3D scene flow) rendered from an initial dynamic 3DGS along input and target cameras, then distills generated virtual-view videos back into the dynamic 3DGS to improve monocular 4D reconstruction.
Problem
Monocular 4D reconstruction with dynamic 3D Gaussians is accurate only in well-observed regions and struggles with novel views, unobserved content, and unseen dynamics, while video generators lack multiview consistency.
Approach
Condition a video generator on pixel-aligned 4D buffers (appearance, geometry, and 3D scene flow) rendered from a dynamic 3DGS along both input and target trajectories, add reconstruction classifier-free guidance, and re-optimize the 3DGS (including motion) on the generated virtual-view videos.
Key results
- On DyCheck (static-camera, covisible), WT + Ours (distill) reaches 19.96 mPSNR, 0.724 mSSIM, outperforming baselines such as ViDAR (19.44 mPSNR).
- Against generative methods on DyCheck, Ours achieves 19.89 mPSNR and 0.197 mLPIPS versus CAT4D's 18.24 and 0.227.
- On MultiCamVideo, Ours (sample) reaches 27.43 PSNR and 0.083 LPIPS versus ViDAR's 22.14 PSNR and 0.198 LPIPS.
Surprising findings
- The distilled 3DGS is worse than the generated samples in PSNR for the MoSca backbone (17.36 distill vs 18.43 sample on DyCheck full), indicating distillation does not always improve pixel metrics.
- Reducing the video generator from 14B to 1.3B degrades performance (PSNR 19.50 to 18.64), suggesting benefits from larger generators.
Limitations
- Fails when the initial scene reconstruction fails completely, and artifacts from the input can propagate to the output.
- Struggles with fluids, smoke, and volumetric stochastic effects.
- Alignment with input degrades when the target trajectory moves far from the input trajectory.
Experimental setup
Datasets: MultiCamVideo DyCheck
Benchmarks: DyCheck MultiCamVideo
Metrics: PSNR SSIM LPIPS mPSNR mSSIM mLPIPS PCK@0.05
Baselines: Shape of Motion MoSca WorldTree ViDAR CAT4D ReCamMaster TrajectoryCrafter GEN3C Vista4D
Builds on: Wan 2.1 VACE Shape of Motion WorldTree MoSca ReCamMaster APG
Closest papers by meaning
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.91)
- FreeTimeGS: Free Gaussian Primitives at Anytime Anywhere for Dynamic Scene Reconstruction (CVPR 2025 · similarity 0.90)
- Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction (NeurIPS 2025 · similarity 0.90)
- MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors (ICLR 2025 · similarity 0.90)
- Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses (CVPR 2024 · similarity 0.90)
- ProDyG: Progressive Dynamic Scene Reconstruction via Gaussian Splatting from Monocular Videos (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.