Research map / Video / Video generation
Controlling Space and Time with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4DiM, a cascaded pixel-diffusion model for 4D novel view synthesis that jointly conditions on camera pose and timestamps, trained on a mix of posed 3D, posed 4D, and unposed video data.
Problem
Generating multi-view consistent scenes with free-form camera pose and time control from one or more images, where posed 3D/4D data is scarce and SfM poses have unknown scale.
Approach
Masked FiLM layers that become identity when a conditioning signal (pose, time, or image) is missing, enabling co-training on posed 3D, 4D, and unposed video; multi-guidance generalizes classifier-free guidance to separate weights per conditioning variable; and a scale-calibrated RealEstate10K using monocular metric depth.
Key results
- 8-frame 4DiM-R (cRE10K-only) achieves FID 31.23 vs 37.35 for ZeroNVS and 51.41 for PNVS on cRE10k test, and PSNR 18.09 vs 15.30 for ZeroNVS.
- Calibrated cRE10K improves zero-shot LLFF FID from 71.07 to 63.48 and TSED from 0.7727 to 0.9659 relative to uncalibrated RE10K training.
- On frame interpolation on DAVIS, 4DiM achieves FID 52.89, better than LDMVFI (56.28), RIFE (57.68), and FILM (68.88).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on calibrated cRE10k instead of uncalibrated RE10k (ablation) | 4DiM-R · RE10k test | FID: 32.40 -> 31.23 | Helped |
| train on calibrated cRE10k instead of uncalibrated RE10k (ablation) | 4DiM-R · LLFF zero-shot | FID: 71.07 -> 63.48 | Helped |
| train on calibrated cRE10k instead of uncalibrated RE10k (ablation) | 4DiM-R · ScanNet++ zero-shot | FID: 23.68 -> 22.89 | Helped |
| add video co-training (ablation) | 4DiM · cRE10k test | FID: 32.45 -> 31.96 | Helped |
| add video co-training (ablation) | 4DiM · ScanNet++ test | FID: 24.61 -> 23.48 | Helped |
| add video co-training (ablation) | 4DiM · cLLFF zero-shot | FID: 64.34 -> 63.78 | Helped |
| filter scenes with unreliable scale calibration (ablation) | 4DiM-R · cRE10k test | FID: 31.81 -> 31.23 | Helped |
| filter scenes with unreliable scale calibration (ablation) | 4DiM-R · cLLFF zero-shot | FID: 62.55 -> 63.48 | Hurt |
| increase conditioning frames from 1 to 8 (ablation) | 4DiM · 30M video dataset test split | FID: 62.73 -> 43.34 | Helped |
| increase conditioning frames from 1 to 6 (ablation) | 4DiM · Street View panoramas | FID: 119.7 -> 37.43 | Helped |
| increase conditioning frames from 1 to 6 (ablation) | 4DiM · Matterport3D 360° panoramas | FID: 86.41 -> 24.86 | Helped |
| use 4DiM instead of LDMVFI for frame interpolation | DAVIS frame interpolation · DAVIS frame interpolation | FID: 56.28 -> 52.89 | Helped |
Surprising findings
- Training without video performs slightly better in-distribution on RealEstate10K but generalizes worse out-of-distribution, where it infers incorrect geometry on LLFF.
- PNVS achieves the best TSED on LLFF, but this is attributed to ~3x fewer SIFT keypoint matches, making the score unreliable.
Limitations
- Single-image conditioning produces little motion; dynamics require two or more input frames.
- Limited 360-degree rotation fidelity and image quality, attributed to model size and calibrated data scale.
- Reconstruction metrics penalize plausible generations and favor blur, and TSED can be unreliable with few keypoint matches.
Experimental setup
Datasets: RealEstate10K ScanNet++ Matterport3D Street View 30M videos LLFF DAVIS CO3D MVImgNet
Benchmarks: RealEstate10K LLFF ScanNet++ DAVIS Street View panoramas Matterport3D panoramas
Metrics: FID FDD FVD TSED SfMD LPIPS PSNR SSIM keypoint distance
Baselines: PNVS MotionCtrl ZeroNVS LDMVFI FILM RIFE
Builds on: 3DiM classifier-free guidance InstructPix2Pix UViT v-parametrization
Closest papers by meaning
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.90)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.88)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.88)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.