Research map / 3D vision / 3D generation
Novel View Synthesis with Pixel-Space Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts an EDM2-style pixel-space cascaded diffusion model with a source-view encoder and cross-view attention (VIVID) for single-image novel view synthesis, and adds a homography-based rotation augmentation to train on single-view data.
Problem
Single-image novel view synthesis on scene-level data, where prior generative NVS methods rely on latent diffusion, complex geometry encodings, and multi-view datasets that are limited in scene diversity and scale accuracy.
Approach
A pixel-space two-U-Net diffusion model (source encoder + target decoder with joint self/cross-view attention) using only simple pose scalar embedding, plus a homography-based camera-rotation augmentation that lets single-view images be used for NVS training.
Key results
- VIVID achieves FID 2.89 and PSNR 17.36 on mid-range RealEstate10K pairs versus GenWarp's 5.91 and 13.43, a substantial improvement over prior SOTA.
- On long-range RealEstate10K pairs VIVID reaches FID 3.89 and PSNR 15.21, compared with GenWarp's 7.38 and 12.10.
- Mixing 10% single-view OpenImages data with homography augmentation lowers out-of-domain FID from 36.14 to 31.98 and raises PSNR from 14.38 to 14.52, while 25% degrades performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add scalar pose embedding instead of using no geometry encoding (ablation) | VIVID · RealEstate10K 64×64 test pairs | FID: 5.75 ± 0.04 -> 3.00 ± 0.02 | Helped |
| Add scalar pose embedding instead of using no geometry encoding (ablation) | VIVID · RealEstate10K 64×64 test pairs | PSNR: 13.39 ± 0.02 -> 21.11 ± 0.04 | Helped |
| Add epipolar attention to pose embedding (ablation) | VIVID · RealEstate10K 64×64 test pairs | FID: 3.00 ± 0.02 -> 2.87 ± 0.04 | Helped |
| Add epipolar attention to pose embedding (ablation) | VIVID · RealEstate10K 64×64 test pairs | PSNR: 21.11 ± 0.04 -> 21.15 ± 0.04 | No clear effect |
| Add monocular depth input to pose embedding (ablation) | VIVID · RealEstate10K 64×64 test pairs | FID: 3.00 ± 0.02 -> 2.99 ± 0.01 | No clear effect |
| Add monocular depth input to pose embedding (ablation) | VIVID · RealEstate10K 64×64 test pairs | PSNR: 21.11 ± 0.04 -> 21.29 ± 0.04 | Helped |
| Add warped coordinate embeddings to pose embedding (ablation) | VIVID · RealEstate10K 64×64 test pairs | FID: 3.00 ± 0.02 -> 3.08 ± 0.04 | Hurt |
| Add warped coordinate embeddings to pose embedding (ablation) | VIVID · RealEstate10K 64×64 test pairs | PSNR: 21.11 ± 0.04 -> 21.06 ± 0.04 | No clear effect |
| Train with 10% single-view OpenImages data using homography augmentation instead of 0% (ablation) | VIVID · 20 out-of-domain scenes | FID: 36.14 -> 31.98 | Helped |
| Train with 10% single-view OpenImages data using homography augmentation instead of 0% (ablation) | VIVID · 20 out-of-domain scenes | PSNR: 14.38 -> 14.52 | Helped |
| Increase single-view OpenImages training data from 10% to 25% (ablation) | VIVID · 20 out-of-domain scenes | FID: 31.98 -> 35.27 | Hurt |
Surprising findings
- Sophisticated geometry encodings (epipolar attention, monocular depth, coordinate warping) give only minor gains over simple pose scalar embedding.
- Latent diffusion via a Stable Diffusion autoencoder loses fine source details that cannot be recovered, even when the model is otherwise strong.
Limitations
- Homography augmentation simulates only camera rotations and no translation, so trajectories rare in training data may not generalize.
- Dependence on scene scales in training data, which can be problematic when SfM-derived scales are not normalized.
- Diverse samples can occasionally produce unrealistic features in large camera transformations.
Experimental setup
Datasets: RealEstate10K OpenImages v5
Benchmarks: RealEstate10K LLFF MipNeRF-360 Ref-NeRF
Metrics: FID PSNR
Baselines: GeoGPT GenWarp Photometric-NVS (PhotoNVS) Source view copy
Builds on: EDM2 cascaded diffusion models GeoGPT GenWarp Pose-Guided Diffusion
Closest papers by meaning
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.92)
- NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer (ICLR 2025 · similarity 0.91)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.91)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.90)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.90)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.