Research map / Video / Video generation
World-consistent Video Diffusion with Explicit 3D Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WVD, a 2B diffusion transformer that jointly models RGB and XYZ (global coordinate) frames as 6D videos, enabling single-image 3D, multi-view stereo, depth estimation, and camera-controlled video generation via inpainting.
Problem
Multi-view and video diffusion models generate 3D-consistent content only implicitly, lacking explicit 3D guarantees and struggling to scale with camera-conditioned designs.
Approach
Represent 3D geometry as normalized XYZ images that share the RGB image shape, concatenate RGB and XYZ latents along channels, and fine-tune a pretrained video diffusion transformer on the joint 6D video distribution, using inpainting to perform multiple 3D tasks without camera conditioning.
Key results
- WVD achieves KPM 95.8 and FC 95.4 on single-image-to-3D versus CameraCtrl's 88.6 and 94.0, though its FID (15.8) is worse than CameraCtrl's 12.1.
- On BONN monocular depth, WVD reaches AbsRel 7.0 and delta1.25 96.4, beating all listed zero-shot baselines including DUSt3R-512 (8.1, 93.6).
- On NYU-v2, WVD gets AbsRel 9.7 vs DUSt3R-512's 6.5 (which uses higher training resolution); WVD removing XYZ raises FID to 18.3 and drops KPM to 72.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly predict XYZ frames together with RGB frames rather than train an RGB-only model (ablation) | WVD · Single image to 3D | FID: 18.3 -> 15.8 | Helped |
| jointly predict XYZ frames together with RGB frames rather than train an RGB-only model (ablation) | WVD · Single image to 3D | KPM: 72.3 -> 95.8 | Helped |
| jointly predict XYZ frames together with RGB frames rather than train an RGB-only model (ablation) | WVD · Single image to 3D | FC: 95.0 -> 95.4 | Helped |
Surprising findings
- Removing XYZ supervision worsens both image quality (FID 18.3 vs 15.8) and multi-view consistency (KPM 72.3 vs 95.8), indicating explicit 3D supervision helps even RGB generation.
Limitations
- Trained only on static datasets, limiting application to static scenes.
- No confidence maps, making unbounded or outdoor scenes challenging.
- Trained at 256x256 resolution, lower than some baselines such as DUSt3R-512.
Experimental setup
Datasets: RealEstate10K ScanNet MVImgNet CO3D Habitat NYU-v2 BONN ScanNet++
Benchmarks: NYU-v2 BONN ScanNet++
Metrics: FID KPM FC AbsRel delta1.25 delta1.03
Baselines: CameraCtrl MotionCtrl DUSt3R RobustMIX SlowTv COLMAP DeepV2D Robust MVD
Builds on: DiT latent diffusion models CAT3D DUSt3R
Closest papers by meaning
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.89)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.89)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.88)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.88)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.88)
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.