Research map / 3D vision / 3D generation
Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diffusion² composes the scores of pretrained video (SVD) and multi-view (SV3D) diffusion models to jointly sample a dense multi-view multi-frame image matrix, then reconstructs a dynamic Gaussian 4D representation, without any 4D training data.
Problem
4D (dynamic 3D) content generation is hard because synchronized multi-view video data is scarce, and existing SDS-based methods are slow and unstable.
Approach
Approximate the score of the full view-by-frame image matrix as the sum of the row score from a video model and the column score from a multi-view model minus the pixel score (conditional independence assumption), and sample the matrix in one reverse diffusion pass with variance-reducing sampling via interpolated re-noising steps to reconcile the two models' heterogeneous scores.
Key results
- On video-to-4D, the method reports CLIP 0.922, LPIPS 12.35 and FVD 1304.18, the best of the compared methods (Consistent4D, STAG4D, DG4D, Efficient4D).
- Generation completes in about 12 minutes for video-to-4D and within 20 minutes in the stated setting on 8 NVIDIA A6000 GPUs, much faster than Consistent4D (120 min) and STAG4D (90 min).
- In a 50-participant user study on image-to-4D, the method was preferred for reference consistency (88.3%), multi-view consistency (58.9%), temporal smoothness (43.1%) and overall quality (59.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diffusion² instead of STAG4D | Video-to-4D generation · Consistent4D test split | CLIP score: 0.920 -> 0.922 | Helped |
| Use Diffusion² instead of STAG4D | Video-to-4D generation · Consistent4D test split | LPIPS: 12.78 -> 12.35 | Helped |
| Use Diffusion² instead of Efficient4D | Video-to-4D generation · Consistent4D test split | FVD: 1381.32 -> 1304.18 | Helped |
| Use Diffusion² instead of DG4D | Video-to-4D generation · Video-to-4D | Generation time: 11 mins -> 12 mins | Hurt |
| Use Diffusion² instead of STAG4D | Image-to-4D generation · Image-to-4D user study | Ref. consistency preference: 5.5% -> 88.3% | Helped |
| Use Diffusion² instead of STAG4D | Image-to-4D generation · Image-to-4D user study | Multi-view consistency preference: 22.9% -> 58.9% | Helped |
| Use Diffusion² instead of STAG4D | Image-to-4D generation · Image-to-4D user study | Temporal smoothness preference: 36.2% -> 43.1% | Helped |
| Use Diffusion² instead of STAG4D | Image-to-4D generation · Image-to-4D user study | Overall model quality preference: 18.2% -> 59.6% | Helped |
| Add variance-reducing sampling (VRS) (ablation) | Diffusion² stage-1 sampling · Multi-view multi-frame image synthesis | Ghosting artifacts | Helped |
| Add score composition of video and multi-view diffusion scores (ablation) | Diffusion² stage-1 sampling · Multi-view multi-frame image synthesis | Spatial-temporal consistency | Helped |
| Replace score composition with independent SV3D generation for each frame (ablation) | Diffusion² stage-1 sampling · Multi-view multi-frame image synthesis | Multi-view consistency | Hurt |
| Replace score composition with spatial-attention feature blending using 1D convolution (ablation) | Diffusion² stage-1 sampling · Multi-view multi-frame image synthesis | Long-range temporal consistency | Hurt |
Limitations
- The conditional independence assumption may not hold strictly, e.g. for large rotations of the object over time.
- Aggressive variance-reducing sampling can sacrifice texture detail in views far from the reference, producing texture-less back views in some cases.
- Parallel denoising trades memory for speed; throughput is not improved and multi-GPU communication limits efficiency.
Experimental setup
Datasets: Consistent4D test split
Benchmarks: Consistent4D test split
Metrics: CLIP score LPIPS FVD generation time user preference
Baselines: Consistent4D STAG4D DreamGaussian4D Efficient4D
Builds on: Stable Video Diffusion SV3D 4D Gaussian splatting (deformable Gaussians) score-based generative models
Closest papers by meaning
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.94)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.94)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.93)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.92)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.91)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.