Research map / Video / Video generation
VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a CogVideoX-5B image-to-video diffusion model, guided by a feed-forward 3DGS (MVSplat) rendering and a learned timestep policy network (DDPNet), into a model that generates 3D-consistent novel-view videos from two input views in one step.
Problem
Video diffusion models used as priors for sparse-view 3D scene generation are slow (dozens of denoising steps) and lack 3D consistency, producing geometry artifacts.
Approach
A 3D-aware leap flow distillation starts consistency distillation from a coarse 3DGS-rendered video encoded and noised at an intermediate timestep (instead of pure noise), combined with a contextual-bandit-trained dynamic denoising policy network that picks the leap timestep at inference.
Key results
- On RealEstate10K, VideoScene with one step reaches FVD 103.42 versus 424.68 for Stable Video Diffusion at 50 steps and 521.04 for CogVideoX-5B at 50 steps.
- On cross-dataset ACID test, VideoScene one-step FVD 121.93 vs. 464.87 for CogVideoX-5B fine-tuned one-step.
- Inference takes about 3s total (about 2.5s for the one-step distilled generation), compared with over 2 minutes for CogVideoX with 50 DDIM steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoScene for one-step generation instead of CogVideoX-5B | Video generation for sparse-view 3D scenes · RealEstate10K | FVD: 753.02 -> 103.42 | Helped |
| Use VideoScene for one-step generation instead of CogVideoX-5B | Video generation for sparse-view 3D scenes · ACID | FVD: 464.87 -> 121.93 | Helped |
| Add 3D-aware leap flow distillation (ablation) | VideoScene · RealEstate10K | FVD: 543.53 -> 97.53 | Helped |
| Add DDPNet for adaptive timestep selection (ablation) | VideoScene · RealEstate10K | FVD: 106.28 -> 97.53 | Helped |
| Replace the full method with the base rendered video (ablation) | VideoScene · RealEstate10K | FVD: 97.53 -> 171.38 | Hurt |
| Add DDPNet for adaptive timestep selection (ablation) | VideoScene · RealEstate10K | Aesthetic Quality: 0.4897 -> 0.5306 | Helped |
Surprising findings
- Baseline video diffusion models degrade sharply at 1 step (e.g., SVD FVD rises from 424.68 at 50 steps to 1220.80 at 1 step), while VideoScene's one-step result stays close to its 50-step result.
Limitations
- Performance is bounded by the underlying backbones MVSplat and CogVideoX.
- Evaluation is limited to two-view input and video-level metrics (FVD, VBench-style scores) rather than downstream 3D reconstruction quality.
Experimental setup
Datasets: RealEstate10K ACID
Benchmarks: RealEstate10K ACID
Metrics: FVD Aesthetic Quality Subject Consistency Background Consistency feature matching count with RANSAC epipolar filtering
Baselines: Stable Video Diffusion DynamiCrafter CogVideoX-5B
Builds on: consistency models latent consistency models CogVideoX MVSplat
Closest papers by meaning
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.87)
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.86)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.86)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.86)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.86)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.