Research map / 3D vision / Gaussian splatting
H3D-DGS: Exploring Heterogeneous 3D Motion Representation for Deformable 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes H3D control points, a heterogeneous 3D motion representation that gets the image-plane motion components from optical flow back-projection and learns only the unobservable depth-direction components by gradient descent, for fast streaming dynamic scene reconstruction with deformable 3DGS.
Problem
Deformable 3D Gaussian Splatting for dynamic multi-view real-world scenes relies on gradient-based optimization of all motion, which has high degrees of freedom and converges poorly on complex real-world motion.
Approach
Decouple local 3D motion into components projected onto the image plane (computed directly from optical flow via back-projection under a local parallel-ray assumption) and unobservable components (learned by gradients), stored as sparse control points that drive nearby Gaussians via K-nearest-neighbor interpolation.
Key results
- On Neu3DV, Ours-GoS5 reaches 33.72 PSNR on sear_steak versus 31.62 for 4D-GS, and per-scene tables show best or near-best PSNR on most sequences.
- Converges within 100 iterations and runs at about 2 seconds per non-keyframe per frame on a single RTX 4070; keyframes take about 10 seconds.
- On CMU-Panoptic softball, Ours-GoS2 reaches 27.48 PSNR versus 26.93 for Dynamic-GS, with better SSIM and LPIPS across the three sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use H3D full motion control instead of no motion control (ablation) | H3D-DGS GoS-10 · Neu3DV sear_steak, first 30 frames | PSNR | Helped |
| Use full H3D motion control instead of projected motion attributes only (ablation) | H3D-DGS GoS-10 · Neu3DV sear_steak, first 30 frames | PSNR | Helped |
| Add keyframe residual compensation (ablation) | H3D-DGS streaming workflow · Neu3DV sear_steak, GoS-10 | PSNR | Helped |
| Add local macro-rotation (ablation) | H3D control points · Dynamic scene reconstruction | reconstruction performance | Hurt |
| Use optical flow estimates with higher accuracy (ablation) | H3D-DGS · Neu3DV | 3D reconstruction quality | Helped |
| Use H3D-DGS instead of the strongest PSNR baseline | Dynamic scene reconstruction · Neu3DV sear_steak, cook_spinach, cut_roasted_beef, flame_steak | PSNR: 31.62 -> 33.72; 32.79 -> 33.20; 32.13 -> 33.23; 30.41 -> 33.18 | Helped |
| Use H3D-DGS instead of the strongest PSNR baseline | Dynamic scene reconstruction · Neu3DV flame_salmon_1 and coffee_martini | PSNR: 28.27 -> 28.00; 28.87 -> 26.90 | Hurt |
| Use H3D-DGS instead of Dynamic-GS | Dynamic scene reconstruction · CMU-Panoptic softball and boxes | PSNR: 26.93 -> 27.48; 27.79 -> 27.88 | Helped |
| Use H3D-DGS instead of Dynamic-GS | Dynamic scene reconstruction · CMU-Panoptic basketball | PSNR: 28.54 -> 27.72 | Hurt |
Surprising findings
- Adding local macro-rotation from ED-graphs reduced performance, so it was omitted from the H3D control points.
Limitations
- Requires multi-view inputs for the initial frames and does not support monocular video.
- Background jitter and floaters arise from the incremental streaming design and accumulated error.
- Reconstruction quality is bounded by the initial static scene reconstruction.
Experimental setup
Datasets: Neu3DV CMU-Panoptic
Benchmarks: Neu3DV CMU-Panoptic
Metrics: PSNR SSIM LPIPS
Baselines: Dynamic-GS MA-GS 4D-GS SP-GS SC-GS
Builds on: 3D Gaussian Splatting Embedded Deformation graph Dynamic 3D Gaussians 4D Gaussian Splatting SC-GS
Closest papers by meaning
- MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.93)
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation (AAAI 2025 · similarity 0.93)
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.92)
- Fully Explicit Dynamic Gaussian Splatting (NeurIPS 2024 · similarity 0.92)
- Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle (CVPR 2024 · similarity 0.92)
- HAIF-GS: Hierarchical and Induced Flow-Guided Gaussian Splatting for Dynamic Scene (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.