Research map / 3D vision / 3D generation
MultiDiff: Consistent Novel View Synthesis from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MultiDiff, a pose-conditional latent video diffusion model that jointly generates multi-view-consistent novel views from one image using depth-warped reference conditioning and structured noise.
Problem
Novel view synthesis / view extrapolation from a single RGB image along a camera trajectory, where autoregressive generation accumulates errors and 3D-based methods are costly and low-resolution.
Approach
Fine-tunes the VideoCrafter video diffusion U-Net with correspondence attention to jointly generate all target frames, conditions it on monocular-depth-warped reference images via a ControlNet-style branch, and samples correlated noise by warping a base Gaussian with the same depth-based warps.
Key results
- On RealEstate10K at 128px, short-term FID improves from 36.37 (DFM) to 25.30 for MultiDiff.
- On ScanNet at 256px, MultiDiff outperforms MVDiffusion on short- and long-term metrics (e.g., short-term FID 43.84 vs 50.18 for MVDiffusion, FVD 114.9 vs 230.1).
- MultiDiff generates a 128px frame in about 1 s versus about 45 s for PhotoNVS and 17 s for DFM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image and video priors instead of training from scratch (ablation) | MultiDiff · RealEstate10K test, 256px | FID (long-term): 85.30 -> 30.15 | Helped |
| add the video diffusion prior (ablation) | MultiDiff · RealEstate10K test, 256px | mTSED: 0.728 -> 0.855 | Helped |
| add warped reference-image conditioning (ablation) | MultiDiff · RealEstate10K test, 256px | mTSED: 0.484 -> 0.855 | Helped |
| add relative camera-pose conditioning (ablation) | MultiDiff · RealEstate10K test, 256px | mTSED: 0.624 -> 0.855 | Helped |
| add structured noise (ablation) | MultiDiff · RealEstate10K test, 256px | PSNR: 15.55 -> 15.65 | Helped |
| add structured noise (ablation) | MultiDiff · ScanNet test, 256px | PSNR: 14.80 -> 15.00 | Helped |
| use MultiDiff instead of PhotoNVS | novel-view synthesis · RealEstate10K test, 256px, long-term | FID: 45.08 -> 30.15 | Helped |
| use MultiDiff instead of PhotoNVS | novel-view synthesis · ScanNet test, 256px, long-term | FID: 51.60 -> 47.11 | Helped |
Surprising findings
- DFM achieves higher PSNR and mTSED on RealEstate10K than MultiDiff despite worse FID, attributed to its PixelNeRF representation.
Limitations
- Relies on off-the-shelf monocular depth estimates that can be noisy and incomplete, causing warping errors.
- DFM-style 3D baselines are limited to 128px, so comparisons are split across resolutions.
- Long-trajectory novel views with little overlap with the reference rely on the pose and embedding conditioning alone and are less informative.
Experimental setup
Datasets: RealEstate10K ScanNet
Benchmarks: RealEstate10K ScanNet
Metrics: PSNR LPIPS FID KID FVD mTSED
Baselines: DFM PhotoNVS Text2Room MVDiffusion
Builds on: VideoCrafter ControlNet DDIM ZoeDepth
Closest papers by meaning
- MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation (CVPR 2024 · similarity 0.90)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.90)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.89)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.88)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.88)
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.