Research map / 3D vision / Gaussian splatting
DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Feed-forward transformer (DGS-LRM) that predicts per-pixel deformable 3D Gaussians with 3D scene flow from a posed monocular video in one pass, trained on a customized multi-view Kubric dataset with ground-truth scene flow.
Problem
Feed-forward reconstruction methods handle only static scenes; predicting deformable 3D representations of dynamic scenes from monocular posed video is hard due to scarce training data with motion ground truth and suitable 3D representations.
Approach
Predict per-pixel deformable Gaussians whose translational deformation vectors warp each Gaussian to any timestamp, trained with dual-view synthetic multi-view Kubric supervision (photometric, depth, and scene flow losses), with temporal tokenization and distant reference frames for efficiency and geometry.
Key results
- On DyCheck with dynamic-object masking, DGS-LRM reaches 11.97 mPSNR in 0.495 s versus L4GM's 5.84 mPSNR at 4.8 s.
- Without masking, DGS-LRM gets 14.89 mPSNR in 0.495 s, close to optimization-based PGDVS (15.88, 3 hours).
- On PointOdyssey, DGS-LRM (FC) attains ATE-3D 0.21 over 120 frames, comparable to SpatialTracker's 0.22.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace L4GM with DGS-LRM | DyCheck DynMask · DyCheck DynMask | mPSNR: 5.84 -> 11.97 | Helped |
| Replace L4GM with DGS-LRM | DyCheck DynMask · DyCheck DynMask | mLPIPS: 0.67 -> 0.51 | Helped |
| Use DGS-LRM with flow chaining instead of SpatialTracker | PointOdyssey · PointOdyssey | ATE-3D: 0.22 -> 0.21 | Helped |
| Add dual-view sampling (ablation) | DGS-LRM training · Kubric holdout | rendering quality | Helped |
| Add scene flow supervision (ablation) | DGS-LRM training · DyCheck | rendering quality | Helped |
| Add distant reference frames (ablation) | DGS-LRM · DyCheck | rendering quality | Helped |
| Use temporal tokenization (ablation) | DGS-LRM · Training and inference | memory consumption | Helped |
| Add real-world videos to the training data (ablation) | DGS-LRM · Real-world reconstruction | reconstruction quality | No clear effect |
| Deform Gaussian attributes other than translation (ablation) | DGS-LRM · Dynamic reconstruction | reconstruction quality | No clear effect |
Surprising findings
- Reconstruction quality is comparable to hours-long optimization methods despite a single 0.495 s feed-forward pass on a 24-frame clip.
- Training on pure synthetic Kubric data generalizes to real-world videos without needing real-world training data, which yielded no major improvement when added.
Limitations
- Cannot handle temporally distant discrete frames or teleporting cameras; requires continuous video with non-stationary camera.
- Scene flow struggles with extremely large motion, partly due to the physically simulated synthetic training distribution.
- Novel-view artifacts grow as the rendered view deviates from the input trajectory.
Experimental setup
Datasets: Kubric (customized MOVi-E setting) DyCheck iPhone DAVIS PointOdyssey
Benchmarks: DyCheck iPhone PointOdyssey DAVIS
Metrics: mPSNR mLPIPS PSNR ATE-3D delta_0.1 delta_0.2 reconstruction time
Baselines: D3DGS PGDVS L4GM SpatialTracker Chained RAFT3D Lifted CoTracker
Builds on: LRM GS-LRM MovieGen temporal autoencoder MegaSaM Kubric
Closest papers by meaning
- Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle (CVPR 2024 · similarity 0.92)
- MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.92)
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation (AAAI 2025 · similarity 0.91)
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (CVPR 2024 · similarity 0.91)
- H3D-DGS: Exploring Heterogeneous 3D Motion Representation for Deformable 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.91)
- 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.