Research map / 3D vision / Gaussian splatting
MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoDGS, a dynamic Gaussian splatting pipeline that renders novel views from casually captured monocular videos, using single-view depth priors, a 3D-aware deformation-field initialization, and an ordinal depth loss.
Problem
Novel-view synthesis of dynamic scenes from casually captured monocular videos with static or slowly moving cameras, where existing dynamic NeRF/3DGS methods fail due to weak multiview constraints.
Approach
Initialize the deformation field in 3D by lifting depth maps and 2D optical flow into 3D flow, and supervise rendered depth with an ordinal depth loss that enforces pixel-pair depth orderings from a single-view depth estimator (GeoWizard) instead of Pearson or scale-invariant depth losses.
Key results
- On DyNeRF, MoDGS reaches 22.64 PSNR / 0.8042 SSIM / 0.1545 LPIPS, versus 19.55 PSNR for Deformable-GS and 15.33 for HexPlane.
- On Nvidia, MoDGS reaches 19.27 PSNR / 0.5235 SSIM / 0.2581 LPIPS, versus 18.07 PSNR for Deformable-GS.
- Ablation on DyNeRF: removing 3D-aware init with ordinal loss gives 21.27 PSNR; 3D-aware init with Pearson loss gives 21.77; full model gives 22.96.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3D-aware deformation initialization instead of random initialization (ablation) | MoDGS · DyNeRF | PSNR: 21.27 -> 22.96 | Helped |
| Use ordinal depth loss instead of Pearson correlation loss (ablation) | MoDGS with 3D-aware initialization · DyNeRF | PSNR: 21.77 -> 22.96 | Helped |
| Add a depth loss rather than remove depth supervision (ablation) | MoDGS · DyNeRF | rendered 3D geometry quality | Helped |
| Set ordinal-loss alpha to 100 rather than 10 (ablation) | MoDGS · DyNeRF flame_steak | PSNR: 21.05 -> 23.23 | Helped |
| Increase ordinal-loss alpha from 100 to 1000 (ablation) | MoDGS · DyNeRF flame_steak | PSNR: 23.23 -> 23.16 | No clear effect |
| Add Gaussian noise Norm(0,0.20) to depth supervision (ablation) | MoDGS · DyNeRF flame_steak | PSNR: 23.23 -> 23.18 | No clear effect |
| Use video-depth priors with ordinal loss instead of Pearson loss (ablation) | MoDGS · DyNeRF sear_steak, cut_beef, coffee_martini | PSNR: 23.43 -> 23.48; 23.19 -> 23.61; 20.01 -> 21.55 | Mixed |
| Replace ordinal depth loss with depth ranking loss (ablation) | MoDGS · DyNeRF sear_steak, cut_beef, coffee_martini | PSNR: 18.22 -> 23.53; 17.07 -> 23.98; 14.09 -> 21.37 | Hurt |
| Replace ordinal depth loss with perceptual LPIPS depth loss (ablation) | MoDGS · DyNeRF sear_steak, cut_beef, coffee_martini | PSNR: 22.43 -> 23.53; 23.36 -> 23.98; 20.97 -> 21.37 | Hurt |
| Replace 3D-aware deformation initialization with zero deformation initialization (ablation) | MoDGS · DyNeRF flame_steak | PSNR: 22.34 -> 23.23 | Hurt |
| Replace 3D-aware deformation initialization with Kaiming initialization (ablation) | MoDGS · DyNeRF flame_steak | PSNR: 22.24 -> 23.23 | Hurt |
| Use video-depth estimator priors instead of GeoWizard priors (ablation) | MoDGS with ordinal depth loss · DyNeRF sear_steak, cut_beef, coffee_martini | PSNR: 23.23 -> 23.48; 23.98 -> 23.61; 21.37 -> 21.55 | Mixed |
Surprising findings
- The paper reports that HexPlane produces unreasonable results on monocular input unless a second camera's video is added for training.
- Existing benchmarks' iPhone dataset poses were judged too inaccurate (epipolar lines miss correspondences), so the authors did not evaluate on it.
Limitations
- Can only reconstruct visible parts and cannot hallucinate occluded or unseen regions.
- Relies heavily on single-view depth estimator quality when the camera is fully static; fails on heavy specular reflections and low light.
- Handles videos up to about 10 seconds; training takes hours per scene.
Experimental setup
Datasets: DyNeRF Nvidia Davis MCV (self-collected in-the-wild videos)
Benchmarks: DyNeRF Nvidia
Metrics: PSNR SSIM LPIPS
Baselines: HexPlane SC-GS Deformable-GS RoDynRF Shape-of-Motion depth ranking loss Pearson correlation depth loss
Builds on: 3D Gaussian Splatting OmniMotion CaDeX RAFT GeoWizard
Closest papers by meaning
- 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization (NeurIPS 2024 · similarity 0.92)
- Efficient Gaussian Splatting for Monocular Dynamic Scene Rendering via Sparse Time-Variant Attribute Modeling (AAAI 2025 · similarity 0.92)
- SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D Gaussians from Monocular Video (CVPR 2025 · similarity 0.91)
- HAIF-GS: Hierarchical and Induced Flow-Guided Gaussian Splatting for Dynamic Scene (NeurIPS 2025 · similarity 0.90)
- Dynamic Gaussian Splatting from Defocused and Motion-blurred Monocular Videos (NeurIPS 2025 · similarity 0.90)
- MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.