Research map / 3D vision / Gaussian splatting
MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MoSca, a 4D motion scaffold of sparse SE(3) trajectory nodes lifted from 2D foundation-model priors, which anchors globally fused dynamic 3D Gaussians for novel-view synthesis from casual monocular videos with unknown camera poses.
Problem
Reconstructing and rendering novel views of dynamic scenes from monocular casual videos with unknown camera parameters is highly ill-posed due to limited multi-view cues and occlusions.
Approach
A sparse graph of SE(3) trajectory nodes (MoSca) with ARAP and smoothness regularization, built from pretrained depth and long-term tracking priors, deforms Gaussians via dual quaternion blending so observations from all frames are globally fused and rendered with Gaussian splatting; camera poses and focal length are solved by tracklet-based bundle adjustment.
Key results
- On DyCheck with camera poses, MoSca reaches 19.32 mPSNR and 0.264 mLPIPS, versus 17.32 mPSNR for Shape-of-Motion.
- On NVIDIA, MoSca achieves 26.72 PSNR and 0.070 LPIPS, the best PSNR among compared methods.
- Camera pose ATE on Sintel is 0.090 versus 0.108 for MonST3R, and on TUM-dynamics 0.031 versus 0.063.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add node control (ablation) | MoSca · DyCheck | mPSNR: 19.28 -> 19.32 | Helped |
| add learnable skinning correction (ablation) | MoSca · DyCheck | mPSNR: 19.27 -> 19.32 | Helped |
| add dual quaternion blending (ablation) | MoSca · DyCheck | mPSNR: 19.18 -> 19.32 | Helped |
| add multi-level topology (ablation) | MoSca · DyCheck | mPSNR: 19.14 -> 19.32 | Helped |
| add geometric optimization stage (ablation) | MoSca · DyCheck | mPSNR: 18.85 -> 19.32 | Helped |
| add photometric optimization stage (ablation) | MoSca · DyCheck | mPSNR: 13.71 -> 19.32 | Helped |
| fuse observations across all frames instead of only 4 neighboring frames (ablation) | MoSca · DyCheck | mPSNR: 16.96 -> 19.32 | Helped |
| fuse observations across all frames instead of only 8 neighboring frames (ablation) | MoSca · DyCheck | mPSNR: 17.26 -> 19.32 | Helped |
| add tracking loss Ltrack (ablation) | MoSca · DyCheck | PCK-T accuracy: 0.737 -> 0.824 | Helped |
| use CoTracker-v3 instead of BootsTAPIR for tracking (ablation) | MoSca · DyCheck | mPSNR: 19.32 -> 19.55 | Helped |
| use Metric3D-v2 instead of LiDAR depth (ablation) | MoSca with BootsTAPIR · DyCheck | mPSNR: 19.32 -> 17.05 | Hurt |
| use UniDepth instead of LiDAR depth (ablation) | MoSca with BootsTAPIR · DyCheck | mPSNR: 19.32 -> 17.12 | Hurt |
Limitations
- Depends on accurate long-term 2D tracks and depth estimates.
- Only reconstructs regions visible at some point in the video; cannot hallucinate never-visible areas.
- Does not model lighting effects such as shadows, reflections, or exposure changes.
Experimental setup
Datasets: DyCheck NVIDIA dynamic video dataset TUM-dynamics Sintel in-the-wild casual videos
Benchmarks: DyCheck NVIDIA dynamic video dataset Sintel TUM-dynamics
Metrics: mPSNR mSSIM mLPIPS PSNR LPIPS ATE RPE trans RPE rot PCK-T
Baselines: Shape-of-Motion Gaussian Marbles 4D GS Dyn.Gaussians HyperNeRF Nerfies DynPoint RoDynRF MonST3R CoTracker
Builds on: 3D Gaussian Splatting Embedded Deformation Graphs Dual Quaternion Blending Shape-of-Motion RoDynRF
Closest papers by meaning
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.89)
- Orientation-anchored Hyper-Gaussian for 4D Reconstruction from Casual Videos (NeurIPS 2025 · similarity 0.89)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.89)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.88)
- 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization (NeurIPS 2024 · similarity 0.88)
- MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splatting (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.