Research map / Video / Video generation
4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4Real, a text-to-4D scene pipeline that generates a reference video and a freeze-time video with a masked Snap Video model, then reconstructs deformable 3D Gaussian splats using reconstruction losses and video SDS.
Problem
Existing text-to-4D generation relies on object-centric multi-view models fine-tuned on synthetic object data, yielding object-centric, non-photorealistic dynamic results that are slow to produce.
Approach
Discard multi-view generative priors and instead use video diffusion models trained on real-world videos: generate a reference video, then a freeze-time video conditioned on one frame to learn a canonical 3DGS with per-frame deformation for inconsistencies, then fit temporal deformation to the reference video, with video SDS in late stages.
Key results
- User study on Amazon MTurk (30 evaluators per pair) shows 4Real preferred over 4Dfy and Dream-in-4D on all seven criteria, e.g. 76.7% vs 23.3% on motion realism against 4Dfy.
- X-CLIP score improves from 20.03 (4Dfy) to 24.23 (Ours) and from 19.52 (Dream-in-4D) to 24.77.
- Generation takes about 1.5 hours on a single A100 GPU versus over 10 hours for competing methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 4Real instead of 4Dfy | Text-to-4D generation · Text-to-4D prompts | X-CLIP: 20.03 -> 24.23 | Helped |
| use 4Real instead of Dream-in-4D | Text-to-4D generation · Text-to-4D prompts | X-CLIP: 19.52 -> 24.77 | Helped |
| use 4Real instead of AYG | Text-to-4D generation · Text-to-4D prompts | X-CLIP: 19.87 -> 23.09 | Helped |
| use 4Real instead of AYG | Text-to-4D generation · Text-to-4D prompts | Visual Quality: 2.49 -> 2.44 | Hurt |
| add per-frame deformation to canonical reconstruction (ablation) | 4Real · Noisy freeze-time videos | Canonical reconstruction quality | Helped |
| add multi-view SDS to canonical reconstruction (ablation) | 4Real · Noisy freeze-time videos | Shape realism | Helped |
| add freeze-time video generation and its reconstruction loss (ablation) | 4Real · Text-to-4D scene reconstruction | Reconstruction success | Helped |
| add joint temporal and multi-view SDS (ablation) | 4Real · Temporal deformation reconstruction | Sharpness | Helped |
| add small motion regularization to per-view deformation (ablation) | 4Real · Freeze-time video reconstruction | Reconstruction quality | Helped |
Surprising findings
- Per-frame deformation is needed to absorb multi-view inconsistencies in generated freeze-time videos; removing it causes visible artifacts on faces and blurred boundaries.
Limitations
- Inherits resolution, blur, and artifact limitations from the underlying video generator, especially for fast motion and thin objects.
- Does not produce high-quality geometry such as meshes due to 3DGS representation.
- Takes over an hour to generate a 2-second 4D scene and fails on abrupt lighting changes or object appearance/disappearance.
Experimental setup
Datasets: Snap Video mixed video datasets (including static real-world object circular-trajectory data)
Benchmarks: 30 text prompts from related works
Metrics: user study win rate X-CLIP score VideoScore visual quality temporal consistency dynamic degree T-V alignment factual consistency
Baselines: 4Dfy Dream-in-4D Align Your Gaussians 3D scene generation + object-centric 4D composition
Builds on: 3D Gaussian Splatting Snap Video score distillation sampling CoGS Dynamic 3DGS
Closest papers by meaning
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.91)
- 4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling (CVPR 2024 · similarity 0.90)
- 4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion (CVPR 2025 · similarity 0.90)
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.90)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.90)
- A Unified Approach for Text- and Image-guided 4D Scene Generation (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.