Research map / Video / Video generation
Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vidu4D reconstructs 4D content from a single generated video using Dynamic Gaussian Surfels with time-varying bone-based warping, warped-state normal regularization, and dual-branch rotation/scale refinement, plus a neural-SDF field initialization.
Problem
Reconstructing high-fidelity, spatially and temporally coherent 4D (sequential 3D) representations from a single generated video, which suffers from non-rigidity, frame distortion, and unknown camera trajectories that break SfM.
Approach
Warps static Gaussian surfels over time with a bone-based dual-quaternion skinning field, regularizes normals in the warped state using a continuous warping field conditioned on coordinate and time, adds a refinement branch for rotation and scaling to reduce flicker, and initializes the warping field from a neural SDF trained with cycle-consistent forward/backward warping.
Key results
- On the 30-video average, DGS reaches 27.30 PSNR, 0.9152 SSIM, 0.0877 LPIPS for novel-view synthesis, versus 24.75 PSNR for SCGS with learned poses.
- On the Cat, Cheetah, and Dragon clips, DGS reaches 24.63, 25.68, and 28.58 PSNR, the best of all compared methods.
- Ablation: removing field initialization drops Cat PSNR from 24.63 to 20.15; removing refinement drops it to 24.19.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DGS instead of SCGS w. Poses | DGS · Average over 30 videos | PSNR: 24.75 -> 27.30 | Helped |
| Use DGS instead of SCGS w. Poses | DGS · Average over 30 videos | SSIM: 0.8680 ± 0.0440 -> 0.9152 ± 0.0602 | Helped |
| Use DGS instead of SCGS w. Poses | DGS · Average over 30 videos | LPIPS: 0.1201 ± 0.0359 -> 0.0877± 0.0564 | Helped |
| Add field initialization (ablation) | DGS · Cat | PSNR: 20.15 -> 24.63 | Helped |
| Add field initialization (ablation) | DGS · Cheetah | PSNR: 20.96 -> 25.68 | Helped |
| Add field initialization (ablation) | DGS · Dragon | PSNR: 25.33 -> 28.58 | Helped |
| Add rotation and scaling refinements (ablation) | DGS · Cat | PSNR: 24.19 -> 24.63 | Helped |
| Add rotation and scaling refinements (ablation) | DGS · Cheetah | PSNR: 24.10 -> 25.68 | Helped |
| Add rotation and scaling refinements (ablation) | DGS · Dragon | PSNR: 27.71 -> 28.58 | Helped |
| Add warped-state normal regularization (ablation) | DGS · 4D reconstruction | surface structural integrity | Helped |
| Add rotation and scaling refinements (ablation) | DGS · 4D reconstruction | fine-grained appearance detail | Helped |
| Add rotation and scaling refinements (ablation) | DGS · 4D reconstruction | texture flickering | Helped |
Limitations
- Reliance on the quality of the input generated video
- Scalability challenges for large scenes
- Computational cost, which limits real-time applications
Experimental setup
Datasets: generated videos from Vidu (text-to-video)
Benchmarks: generated videos (Cat, Cheetah, Dragon; 30-video average)
Metrics: PSNR SSIM LPIPS
Baselines: BANMo D-NeRF Deformable-GS SCGS
Builds on: 3D Gaussian splatting 2D Gaussian surfels BANMo NeuS dual quaternion blend skinning
Closest papers by meaning
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.89)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.89)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.88)
- DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation (NeurIPS 2024 · similarity 0.87)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.87)
- Orientation-anchored Hyper-Gaussian for 4D Reconstruction from Casual Videos (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.