Research map / Video / Video generation
SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpatialDreamer, a self-supervised video diffusion framework that turns monocular video into stereo video using depth-based forward-backward rendering data generation, a RefinerNet reference branch, and consistency control (stereo deviation strength and temporal interaction learning).
Problem
Monocular-to-stereo video synthesis suffers from a lack of paired stereo video training data and from geometric and spatio-temporal inconsistency across frames when NVS image methods are applied to video.
Approach
Self-generate paired stereo videos via depth-based rendering with optical-flow-refined occlusion masks, then train an SVD-based video diffusion model with a RefinerNet (SD2.1-initialized spatial reference branch), a stereo deviation strength conditioning with stereo-aware loss, and an attention-based temporal interaction module over adjacent frames.
Key results
- On RealEstate10K (t=5), the method reaches SSIM 0.916, PSNR 32.26, LPIPS 0.038, beating SinMPI (0.898, 28.79, 0.040).
- On the stereo video benchmark, FVD is 67.09 versus 83.18 for webui-depthmap and 99.92 for AVP, with Ewarp 3.374e-3 (best).
- Ablation shows FVD dropping from 184.0 (base) to 67.09 with TIL and DVG added.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RefinerNet instead of using U-Net features only (ablation) | SpatialDreamer · RealEstate10K stereo image | PSNR: 23.73 -> 30.20 | Helped |
| replace RefinerNet with ControlNet (ablation) | SpatialDreamer · RealEstate10K stereo image | LPIPS: 0.043 -> 0.183 | Hurt |
| add stereo deviation strength (SDS) (ablation) | SpatialDreamer with RefinerNet · RealEstate10K stereo image | PSNR: 30.20 -> 32.26 | Helped |
| add Temporal Interaction Learning (TIL) (ablation) | SpatialDreamer video model · Stereo video benchmark | FVD: 184.0 -> 123.5 | Helped |
| add Depth-based Video Generation (DVG) (ablation) | SpatialDreamer video model · Stereo video benchmark | FVD: 184.0 -> 85.21 | Helped |
| add TIL and DVG (ablation) | SpatialDreamer video model · Stereo video benchmark | FVD: 184.0 -> 67.09 | Helped |
| use SpatialDreamer instead of SinMPI | Stereo image synthesis · RealEstate10K t=5 | PSNR: 28.79 -> 32.26 | Helped |
| use SpatialDreamer instead of SinMPI | Stereo image synthesis · RealEstate10K t=10 | LPIPS: 0.080 -> 0.049 | Helped |
| use SpatialDreamer instead of AVP | Stereo video synthesis · Stereo video benchmark | FVD: 99.92 -> 67.09 | Helped |
| use SpatialDreamer instead of 3D-photography | Stereo video synthesis · Stereo video benchmark | Ewarp (×10−3): 3.418 -> 3.374 | Helped |
Limitations
- Relies on depth maps for rendering, so stereo quality depends on depth accuracy and distribution.
- Large parameter size limits real-time performance.
- No open-source stereo video benchmark exists; evaluation relies on a self-collected set.
Experimental setup
Datasets: RealEstate10K self-collected monocular videos (1500 total, 1400 train, 100 test)
Benchmarks: RealEstate10K self-collected stereo video test set
Metrics: SSIM PSNR LPIPS FVD flow warping error Ewarp
Baselines: 3D-photography SynSin AdaMPI SinMPI Wang et al. Photoconsistent-NVS NVSVDE-Net CoPoNeRF MVSplat webui-depthmap NVS-Solver AVP
Builds on: Stable Video Diffusion Stable Diffusion 2.1 ControlNet MiDaS
Closest papers by meaning
- StereoWorld: Geometry-Aware Monocular-to-Stereo Video Generation (CVPR 2026 · similarity 0.90)
- SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix (ICLR 2025 · similarity 0.86)
- Geometry-guided Online 3D Video Synthesis with Multi-View Temporal Consistency (CVPR 2025 · similarity 0.86)
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.86)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.85)
- Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.