Research map / Video / Video generation
4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4Real-Video, a feedforward two-stream diffusion transformer that generates 4D video grids (time x viewpoint) by synchronizing row-wise temporal and column-wise view token streams with hard or soft sync layers.
Problem
Generating consistent 4D videos (a grid of frames with time and viewpoint axes) that are photorealistic and generalize beyond the small amount of available synthetic 4D data, faster than optimization-based methods.
Approach
Split video tokens into two parallel streams (temporal row updates and view column updates) using pretrained DiT blocks, and couple them after each layer with a synchronization layer (hard: learned weighted merge; soft: modulated linear predicting separate residual updates), trained first on 2D-transformed pseudo-4D videos and then fine-tuned on animated Objaverse with only sync layers updated.
Key results
- Soft Sync reduces time-axis FVD to 906.16 vs 1392.48 for soft w/o Objaverse and 1662.54 for sequential interleaving (Table 1).
- Soft Sync achieves Dust3R-Confidence of 47.1 at tau=0.1 and 41.0 at tau=2.0 versus 28.6 and 33.5 for sequential (Table 2).
- User study with 10 evaluators per pair shows 4Real-Video outperforms 4Dfy, Dream-in-4D, AYG and 4Real across all seven criteria by a large margin (Fig. 8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sequential interleaving with the parallel two-stream architecture (ablation) | 4Real-Video · 4D video test set | FVD: 1662.54 -> 906.16 | Helped |
| replace hard synchronization with soft synchronization (ablation) | 4Real-Video · 4D video test set | FVD: 972.87 -> 906.16 | Helped |
| add Animated Objaverse fine-tuning after pseudo-4D pretraining (ablation) | 4Real-Video · 4D video test set | FVD: 1392.48 -> 906.16 | Helped |
| add the parallel two-stream architecture instead of sequential interleaving (ablation) | 4Real-Video · Fixed-view videos | GIM-Confidence (τ=0.1): 65.0 -> 79.6 | Helped |
| replace hard synchronization with soft synchronization (ablation) | 4Real-Video · Fixed-view videos | Dust3R-Confidence (τ=2.0): 39.3 -> 41.0 | Helped |
| add Animated Objaverse fine-tuning after pseudo-4D pretraining (ablation) | 4Real-Video · Fixed-view videos | Dust3R-Confidence (τ=2.0): 39.1 -> 41.0 | Helped |
| replace MotionCtrl with 4Real-Video | 4Real-Video · 4D video test set | FVD: 1556.36 -> 906.16 | Helped |
| replace SV4D with 4Real-Video | 4Real-Video · 4D video test set | CLIP: 19.46 -> 28.22 | Helped |
Surprising findings
- Training without any 4D data (pseudo-4D only) yields competitive results versus baselines.
- Mixing pseudo-4D and Objaverse data in joint training did not beat sequential pretraining then fine-tuning; pseudo data distracted the model toward 2D transforms.
Limitations
- Small 600M base model limits visual quality and resolution.
- No support for 360-degree video generation and no camera pose conditioning.
- Freeze-time video generation is weak for dynamic elements such as running horses or fire.
Experimental setup
Datasets: Animated Objaverse pseudo-4D videos from 2D affine transformations Snap Video generated test pairs
Benchmarks: 200 Snap Video-generated freeze-time/fixed-view test pairs
Metrics: VideoScore FVD FVD-Test CLIP score GIM-Confidence Dust3R-Confidence user study win rate
Baselines: MotionCtrl SV4D 4Real 4Dfy Dream-in-4D AYG Sequential interleaving
Builds on: DiT 4Real rectified flow Snap Video
Closest papers by meaning
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.90)
- Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation (NeurIPS 2025 · similarity 0.89)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.89)
- SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints (ICLR 2025 · similarity 0.89)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.