Research map / 3D vision / Gaussian splatting
DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamMesh4D, a video-to-4D framework that binds Gaussians to a mesh (SuGaR-style hybrid) and deforms it with sparse-control deformation graph and a learned LBS/DQS adaptive hybrid skinning, outperforming Consistent4D, DreamGaussian4D, 4DGen and STAG4D.
Problem
Generating high-quality, spatio-temporally consistent dynamic 3D objects (4D) from monocular videos, where NeRF and Gaussian splatting representations lack surface constraints and yield poor surface appearance.
Approach
Combine a Gaussian-mesh hybrid (SuGaR-style Gaussians bound to mesh faces, optimized jointly with mesh vertices) with sparse-control deformation graph using geodesic neighbors and an adaptive blend of LBS and DQS per vertex governed by a predicted local rigid strength.
Key results
- Achieves PSNR(ref) 37.04 and SSIM 0.980 versus best baseline DreamGaussian4D 31.06 and 0.947 on the Consistent4D test set.
- Reports FVD 474.96 versus 913.10 for the best baseline 4DGen (16 frames), roughly half, and lowest FID-VID 29.14 vs 31.84 for Consistent4D.
- Adaptive hybrid skinning improves over LBS-only (FVD 540.86) and DQS-only (FVD 479.83) variants, with full method at 474.96.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DreamMesh4D instead of the strongest baseline for each metric | Video-to-4D generation on Consistent4D test set · Consistent4D test set | PSNR(ref): 31.06 -> 37.04 | Helped |
| Use DreamMesh4D instead of the strongest baseline for each metric | Video-to-4D generation on Consistent4D test set · Consistent4D test set | SSIM(ref): 0.947 -> 0.980 | Helped |
| Use DreamMesh4D instead of the strongest baseline for each metric | Video-to-4D generation on Consistent4D test set · Consistent4D test set | LPIPS: 0.133 -> 0.126 | Helped |
| Use DreamMesh4D instead of the strongest baseline for each metric | Video-to-4D generation on Consistent4D test set · Consistent4D test set | FVD: 913.10 -> 474.96 | Helped |
| Use DreamMesh4D instead of the strongest baseline for each metric | Video-to-4D generation on Consistent4D test set · Consistent4D test set | FID-VID: 31.84 -> 29.14 | Helped |
| Use DreamMesh4D instead of the strongest baseline for each metric | Video-to-4D generation on Consistent4D test set · Consistent4D test set | CLIP: 0.917 -> 0.938 | Helped |
| Use adaptive hybrid skinning instead of DQS or LBS (ablation) | DreamMesh4D · Consistent4D test set | LPIPS: 0.126 -> 0.126 (DQS); 0.155 -> 0.126 (LBS) | Mixed |
| Add ARAP loss (ablation) | DreamMesh4D · Consistent4D test set | FVD: 751.56 -> 474.96 | Helped |
| Add normal consistency loss (ablation) | DreamMesh4D · Consistent4D test set | LPIPS: 0.147 -> 0.126 | Helped |
| Increase face Gaussians from 1 to 6 (ablation) | DreamMesh4D · Consistent4D test set | PSNR(ref): 36.17 -> 36.63 | Helped |
| Vary control-node count and vertex connectivity (ablation) | DreamMesh4D · Consistent4D test set | PSNR(ref): 35.92; 35.97; 36.14; 36.28; 36.39; 36.42; 36.70; 37.82 | Mixed |
| Use geodesic rather than Euclidean distance for deformation-graph connections (ablation) | DreamMesh4D · Elephant qualitative case | deformation artifacts | Helped |
Surprising findings
- Training with one GPU (0.8h) is faster than all compared baselines, while using 8GB memory, which is less than Consistent4D and DreamGaussian4D but more than STAG4D.
Limitations
- Relies on Zero123 for novel-view SDS supervision, leading to long optimization and poor performance where Zero123 fails.
- Only handles object-level 4D from fixed-viewpoint videos; scene-level or moving-camera videos are not supported.
- Evaluation limited to a small test set (seven videos), so performance on complex motions is not assessed.
Experimental setup
Datasets: Consistent4D test dataset videos from previous works SVD-generated videos
Benchmarks: Consistent4D test dataset
Metrics: PSNR(ref) SSIM(ref) LPIPS FVD FID-VID CLIP score
Baselines: Consistent4D DreamGaussian4D 4DGen STAG4D
Builds on: SuGaR SC-GS DreamGaussian Zero123 Dual quaternion skinning
Closest papers by meaning
- DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (ICLR 2024 · similarity 0.89)
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.88)
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models (CVPR 2024 · similarity 0.88)
- Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle (CVPR 2024 · similarity 0.88)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.87)
- SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.