Research map / Video / Video generation
4Diffusion: Multi-view Video Diffusion Model for 4D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds 4DM, a multi-view video diffusion model by adding a learnable motion module to a frozen ImageDream, trained on 926 curated Objaverse animations, and uses it with 4D-aware SDS and an anchor loss to optimize hash-encoded dynamic NeRF from monocular video.
Problem
4D generation from monocular video suffers from spatial-temporal inconsistency and flicker because existing pipelines combine multiple diffusion models (3D-aware, image, video) that lack joint multi-view spatial-temporal guidance.
Approach
Insert zero-initialized temporal motion modules into a frozen 3D-aware diffusion model (ImageDream) so one model yields multi-view video priors; use its outputs for a 4D-aware SDS loss and an anchor-video perceptual loss to optimize dynamic NeRF.
Key results
- 4D generation: FVD 417.63 vs 706.07 for Consistent4D and CLIP-C 0.9798 vs 0.9723 (best among compared methods).
- Multi-view video: 4DM reaches FVD 427.34 vs 465.94 for ImageDream and LPIPS 0.1346 vs 0.1536.
- Training on the curated 926-shape subset outperforms training on the whole Objaverse animation set (FVD 427.34 vs 583.79).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 4D-aware SDS, using 4DM rather than ImageDream with 3D-aware SDS (ablation) | 4Diffusion dynamic NeRF · 4D generation test cases | FVD: 1546.4 -> 417.63 | Helped |
| add 4D-aware SDS, using 4DM rather than ImageDream with 3D-aware SDS (ablation) | 4Diffusion dynamic NeRF · 4D generation test cases | CLIP-I: 0.8195 -> 0.9310 | Helped |
| add anchor loss (ablation) | 4Diffusion dynamic NeRF · 4D generation test cases | FVD: 853.57 -> 417.63 | Helped |
| add anchor loss (ablation) | 4Diffusion dynamic NeRF · 4D generation test cases | PSNR: 17.20 -> 19.07 | Helped |
| fine-tune 4DM on the whole Objaverse dataset instead of the curated subset (ablation) | 4DM · Consistent4D test dataset | CLIP-I: 0.9260 -> 0.8872 | Hurt |
| fine-tune 4DM on the whole Objaverse dataset instead of the curated subset (ablation) | 4DM · Consistent4D test dataset | FVD: 427.34 -> 583.79 | Hurt |
| add temporal motion modules to the ImageDream backbone | 4DM · Consistent4D test dataset | CLIP-C: 0.9320 -> 0.9601 | Helped |
| add temporal motion modules to the ImageDream backbone | 4DM · Consistent4D test dataset | CLIP-I: 0.9165 -> 0.9260 | Helped |
Surprising findings
- Fine-tuning the motion module on the entire unfiltered Objaverse animation set degrades image quality below ImageDream, showing data quality matters more than quantity.
Limitations
- Multi-view video model is constrained by the base model capability and the scale of high-quality training data.
- Volumetric rendering makes 4D optimization slow.
- 4DM generates only 8 frames at a time, so longer inputs need multiple passes that can cause temporal inconsistency.
Experimental setup
Datasets: Objaverse Consistent4D dataset Cap3D captions
Benchmarks: Consistent4D dataset Objaverse test cases
Metrics: CLIP-I CLIP-C FVD LPIPS PSNR
Baselines: 4D-fy Consistent4D DreamGaussian4D ImageDream MAV3D
Builds on: ImageDream AnimateDiff 4D-fy DreamFusion Consistent4D
Closest papers by meaning
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.95)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.94)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.92)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.91)
- Animate3D: Animating Any 3D Model with Multi-view Video Diffusion (NeurIPS 2024 · similarity 0.90)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.