Research map / 3D vision / Gaussian splatting
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends 3D Gaussian Splatting to monocular dynamic scenes by learning canonical 3D Gaussians deformed by an MLP deformation field conditioned on time, plus an annealed noise smoothing scheme for pose-induced jitter.
Problem
Reconstructing and rendering monocular dynamic scenes with high fidelity and real-time speed, where implicit neural representations are slow and lose detail, and inaccurate poses cause temporal jitter in time interpolation.
Approach
Jointly optimize 3D Gaussians in canonical space with a purely implicit MLP deformation field (inputs: stop-gradient Gaussian position and time with positional encoding) that outputs offsets to position, rotation and scale, rendered via the differentiable Gaussian rasterizer; an annealed Gaussian noise added to time encoding (AST) smooths training with no extra overhead.
Key results
- On the D-NeRF synthetic scenes, the method achieves the best PSNR/SSIM/LPIPS in the table, e.g., Hell Warrior PSNR 41.54 vs 31.26 for the best baseline Tensor4D.
- On NeRF-DS (mean over seven scenes), the method reaches PSNR 24.11, SSIM 0.8525, LPIPS 0.1769, versus NeRF-DS baseline 23.60 / 0.8494 / 0.1816.
- Real-time rendering above 30 FPS on an RTX 3090 when the number of Gaussians is below 250k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add annealing smooth training (AST) (ablation) | Deformable 3D Gaussians · NeRF-DS | PSNR: 23.97 -> 24.11 | Helped |
| add annealing smooth training (AST) (ablation) | Deformable 3D Gaussians · NeRF-DS | SSIM: 0.8346 -> 0.8525 | Helped |
| add annealing smooth training (AST) (ablation) | Deformable 3D Gaussians · NeRF-DS | LPIPS: 0.2037 -> 0.1769 | Helped |
| add AST noise annealing to time input (ablation) | Deformable 3D Gaussians · real-world time interpolation | temporal smoothness | Helped |
| reduce time positional encoding from the original order to 6pe without AST (ablation) | Deformable 3D Gaussians · real-world time interpolation | preservation of high-frequency information | Hurt |
| use original-order time positional encoding without AST (ablation) | Deformable 3D Gaussians · real-world time interpolation | temporal overfitting | Hurt |
| apply positional encoding to deformation-network inputs (ablation) | Deformable 3D Gaussians · dynamic scene rendering | rendering detail | Helped |
Surprising findings
- Explicit grid/plane structures were deliberately avoided because the low-rank assumption hurts dynamic scenes, with an MLP deformation field preferred instead.
Limitations
- Performance depends on accuracy of COLMAP pose estimation; not optimal PSNR on HyperNeRF.
- Sparse viewpoints can cause overfitting; training time and memory scale with the number of Gaussians.
- Evaluated mainly on moderate motion; complex human motion such as facial expressions untested.
Experimental setup
Datasets: D-NeRF synthetic dataset HyperNeRF NeRF-DS
Benchmarks: D-NeRF synthetic dataset NeRF-DS
Metrics: PSNR SSIM LPIPS FPS
Baselines: 3D-GS D-NeRF TiNeuVox Tensor4D K-Planes HyperNeRF NeRF-DS
Builds on: 3D Gaussian Splatting D-NeRF HyperNeRF positional encoding
Closest papers by meaning
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation (AAAI 2025 · similarity 0.92)
- Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle (CVPR 2024 · similarity 0.92)
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (CVPR 2024 · similarity 0.92)
- Efficient Gaussian Splatting for Monocular Dynamic Scene Rendering via Sparse Time-Variant Attribute Modeling (AAAI 2025 · similarity 0.92)
- COLMAP-Free 3D Gaussian Splatting (CVPR 2024 · similarity 0.91)
- DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.