Research map / Video / Video generation
Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diffusion4D fine-tunes a 3D-aware video diffusion model (VideoMV) on a curated 54K dynamic 3D dataset to generate spatial-temporal consistent orbital videos with motion-magnitude control, then builds 4D Gaussian splatting in a coarse-to-fine manner in minutes.
Problem
Generating 4D content efficiently and with consistent spatial and temporal geometry, as SDS-based optimization is slow and multi-model/multi-view pipelines produce inconsistent geometry.
Approach
Adapt a 3D-aware video diffusion model to output orbital videos of dynamic 3D assets, jointly modeling spatial and temporal consistency in one network, with a 3D-to-4D motion magnitude condition, a motion magnitude reconstruction loss, and 3D-aware classifier-free guidance, followed by explicit 4DGS construction.
Key results
- Text-to-4D: Diffusion4D reaches CLIP-O 0.65 in about 8 minutes versus 23h for 4DFY (CLIP-O 0.61).
- Image-to-4D: Diffusion4D achieves SSIM 0.83, PSNR 16.7, LPIPS 0.21, FVD 560.8, outperforming STAG4D (FVD 675.4) and 4DGen (FVD 736.6).
- User study: 54% overall preference for Diffusion4D in image-to-4D setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diffusion4D instead of 4DFY | Text-to-4D generation · Text-to-4D | Generation time: 23h -> 8m | Helped |
| Use Diffusion4D instead of Animate124 | Text-to-4D generation · Text-to-4D | CLIP-O: 0.58 -> 0.65 | Helped |
| Use Diffusion4D instead of STAG4D | Image-to-4D generation · Image-to-4D | SSIM: 0.76 -> 0.83 | Helped |
| Use Diffusion4D instead of STAG4D | Image-to-4D generation · Image-to-4D | Generation time: 2h30m -> 8m | Helped |
| Add motion magnitude reconstruction loss (ablation) | Diffusion4D image-to-4D generation · Image-to-4D | CLIP-O: 0.73 -> 0.75 | Helped |
| Add motion magnitude reconstruction loss (ablation) | Diffusion4D image-to-4D generation · Image-to-4D | SSIM: 0.78 -> 0.83 | Helped |
| Add coarse construction stage (ablation) | Diffusion4D image-to-4D construction · Image-to-4D | CLIP-O: 0.70 -> 0.75 | Helped |
| Add coarse construction stage (ablation) | Diffusion4D image-to-4D construction · Image-to-4D | FVD: 651.2 -> 560.8 | Helped |
| Add fine construction stage (ablation) | Diffusion4D image-to-4D construction · Image-to-4D | SSIM: 0.77 -> 0.83 | Helped |
| Add fine construction stage (ablation) | Diffusion4D image-to-4D construction · Image-to-4D | FVD: 581.4 -> 560.8 | Helped |
| Increase motion magnitude guidance (ablation) | Diffusion4D 4D-aware diffusion model · Image-to-4D | motion strength | Helped |
| Use 3D-aware classifier-free guidance (ablation) | Diffusion4D 4D-aware diffusion model · Image-to-4D | motion strength | Helped |
Limitations
- Limited resolution (256x256) and 24-frame temporal length.
- Relies on dynamic-labeled assets and only handles orbital views with fixed elevation.
Experimental setup
Datasets: Objaverse-1.0 Objaverse-XL
Benchmarks: curated dynamic 3D test set (20 cases)
Metrics: CLIP-F CLIP-O SSIM PSNR LPIPS FVD generation time
Baselines: 4DFY Animate124 4DGen STAG4D
Builds on: VideoMV 4DGS classifier-free guidance score distillation sampling
Closest papers by meaning
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.95)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.94)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.93)
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models (CVPR 2024 · similarity 0.90)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.90)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.