Research map / 3D vision / Gaussian splatting
4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pose-free dynamic Gaussian splatting from monocular video that uses MonST3R initialization, SAM2-refined motion masks for masked pose refinement (MA-BA), and control-point-driven LBS deformation of Gaussians (MA-GS).
Problem
Novel view synthesis of dynamic scenes from monocular videos with unknown camera poses; existing 4D Gaussian methods depend on precomputed poses, and pose estimators fail when moving objects dominate the image.
Approach
Combine transformer-based dynamic masks (MonST3R) refined with SAM2 point prompts so that only static points are used for PnP-RANSAC/bundle adjustment, and model dynamic motion with a sparse set of control points driving LBS-deformed Gaussians in a two-stage training scheme.
Key results
- On HyperNeRF, achieves 25.6 dB PSNR and 0.844 MS-SSIM, versus 20.4 dB for the MonST3R+SC-GS pose-free baseline and 23.8 dB for RoDynRF.
- On DyNeRF, reports 19.6 dB PSNR and 0.755 MS-SSIM (text cites 0.775), outperforming SC-GS (19.0 dB) with pose-free input.
- On MPI Sintel, ATE of 0.086 is comparable to LEAP-VO (0.089) and better than RoDynRF (0.089), with RPE trans 0.035 matching CasualSAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 4D3R instead of SC-GS | Pose-free dynamic scene rendering · HyperNeRF | PSNR: 25.3 -> 25.6 | Helped |
| use 4D3R instead of SC-GS | Pose-free dynamic scene rendering · DyNeRF | PSNR: 19.0 -> 19.6 | Helped |
| use 4D3R instead of LEAP-VO | Camera pose estimation · MPI Sintel | ATE: 0.089 -> 0.086 | Helped |
| add the motion-aware map (ablation) | 4D3R · HyperNeRF | PSNR: 20.4 -> 25.6 | Helped |
| add SAM2-based mask refinement (ablation) | 4D3R · HyperNeRF | PSNR: 23.8 -> 25.6 | Helped |
| add MA-GS (ablation) | 4D3R · HyperNeRF | PSNR: 23.5 -> 25.6 | Helped |
| add MA-GS (ablation) | 4D3R · HyperNeRF | training time: 1.5 hours -> 50mins | Helped |
Surprising findings
- Ablation without the motion map drops PSNR by 5.2 dB (text) while the table reports the same 20.4 dB as the MonST3R+SC-GS baseline, suggesting inconsistent reporting.
- Reported PSNR of 25.6 dB on HyperNeRF is marginally above 4DGS and SC-GS, which use COLMAP poses, despite being pose-free.
Limitations
- Requires textured frames with sufficient static regions and distinguishable dynamic objects.
- Struggles with complex non-rigid deformations.
- Relies on pretrained models (MonST3R, SAM2) for masks and initialization.
Experimental setup
Datasets: HyperNeRF DyNeRF MPI Sintel
Benchmarks: HyperNeRF DyNeRF MPI Sintel
Metrics: PSNR SSIM MS-SSIM ATE RPE trans RPE rot FPS storage training time
Baselines: 3D-GS Nerfies HyperNeRF TiNeuVox 4DGS FDNeRF SC-GS MonST3R+SC-GS RoDynRF DROID-SLAM DPVO ParticleSFM
Builds on: 3D Gaussian Splatting MonST3R DUSt3R SC-GS SAM2 SEA-RAFT
Closest papers by meaning
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.91)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.91)
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation (AAAI 2025 · similarity 0.91)
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (CVPR 2024 · similarity 0.90)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.90)
- Flux4D: Flow-based Unsupervised 4D Reconstruction (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.