Research map / Image generation / Text-to-image diffusion
A Unified Approach for Text- and Image-guided 4D Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Dream-in-4D generates text-, image-, or multi-image-guided 4D scenes in two stages: a static NeRF guided by MVDream (3D) and StableDiffusion (2D), then a frozen-canonical deformable NeRF with hash-grid deformation and a displacement TV loss optimized with Zeroscope video SDS.
Problem
Text-to-4D dynamic 3D scene generation with diffusion guidance, where prior work (MAV3D) suffers from the Janus problem, low texture quality, and entangled geometry and motion, and cannot handle image- or personalized-conditioned 4D generation.
Approach
Combine 3D-aware (MVDream), 2D (StableDiffusion) and video (Zeroscope) diffusion priors in a two-stage pipeline, with a D-NeRF representation that freezes the learned static canonical asset while optimizing a multi-resolution hash-grid deformation field under video SDS plus a total variation loss on rendered displacements; swapping the static-stage diffusion models enables image-to-4D and personal
Key results
- Against MAV3D, users preferred Dream-in-4D 82.4% of the time for visual quality and 3D consistency, 65.4% for text alignment, and 61.8% for motion.
- Removing 2D guidance in the static stage drops text alignment preference to 11.67% versus 88.33% for the full method.
- Without the TV loss, motion quality preference falls to 2.78% against 55.00% for the full method (42.22% for the version without multi-resolution grid).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Dream-in-4D instead of MAV3D | Text-to-4D generation · Text-to-4D user study | visual quality and 3D consistency preference: 82.4% | Helped |
| use Dream-in-4D instead of MAV3D | Text-to-4D generation · Text-to-4D user study | text alignment preference: 65.4% | Helped |
| use Dream-in-4D instead of MAV3D | Text-to-4D generation · Text-to-4D user study | motion quality preference: 61.8% | Helped |
| add 2D diffusion guidance (ablation) | Dream-in-4D static stage · Text-to-4D user study | text alignment preference: 11.67% -> 88.33% | Helped |
| add a multi-resolution feature grid for deformation (ablation) | Dream-in-4D motion learning · Text-to-4D user study | motion quality preference: 42.22% -> 55.00% | Helped |
| add total variation loss on displacement (ablation) | Dream-in-4D motion learning · Text-to-4D user study | motion quality preference: 2.78% -> 55.00% | Helped |
| add 3D diffusion guidance (ablation) | Dream-in-4D static stage · Text-to-4D user study | visual quality and 3D consistency preference: 6.12% -> 93.88% | Helped |
| use a motion-disentangled deformable NeRF instead of Hexplane (ablation) | Dream-in-4D motion representation · Text-to-4D user study | visual quality and 3D consistency preference: 0.00% -> 93.88% | Helped |
| use Dream-in-4D instead of MAV3D | Image-to-4D generation · Image-to-4D user study | image texture preservation preference: 98.3% | Helped |
| use Dream-in-4D instead of MAV3D | Image-to-4D generation · Image-to-4D user study | text alignment preference: 100.0% | Helped |
Surprising findings
- Relying solely on image or video diffusion guidance for static text-to-3D synthesis easily yields the Janus problem, and fine-tuning the static model with video diffusion lowers visual quality and prompt fidelity.
Limitations
- Fails on difficult prompts where the static stage produces wrong or view-inconsistent 3D assets (e.g., a robot playing violin).
- The dynamic stage cannot correct errors inherited from the static stage, such as incorrect hand positions.
- Evaluation relies on small user studies (13 and 18 users) and no public MAV3D implementation, so comparison is against 28 website videos.
Experimental setup
Metrics: user preference percentage
Baselines: MAV3D w/o 2D guidance w/o 3D guidance Hexplane representation w/o multi-res grid w/o TV loss
Builds on: DreamFusion MVDream D-NeRF MAV3D Instant-NGP
Closest papers by meaning
- 4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling (CVPR 2024 · similarity 0.90)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models (CVPR 2024 · similarity 0.89)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.88)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.