Research map / 3D vision / 3D generation
Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MVGD, a pixel-level multi-task diffusion transformer (RIN-based) that jointly generates novel-view images and depth maps from an arbitrary number of posed views without an intermediate 3D representation, trained on 60M+ multi-view samples.
Problem
Generalizable novel view synthesis and depth estimation from sparse posed images typically rely on intermediate 3D representations (NeRFs, voxels, 3DGS); generating multi-view consistent novel views and depth directly with diffusion is hard.
Approach
A diffusion model using Recurrent Interface Networks operating directly in pixel space, with raymap conditioning, learnable task embeddings for joint RGB and depth generation, scene scale normalization for scale-aware depth, and incremental latent-token duplication to scale up models.
Key results
- With 2 input views, MVGD reaches 28.41 PSNR on RealEstate10K and 29.98 on ACID, outperforming MVSplat (26.39 and 28.25).
- On ScanNet stereo depth with 2 views, MVGD achieves Abs.Rel 0.065 versus 0.086 for the prior best (EPIO).
- Incremental fine-tuning from 256 to 2048 latent tokens improves RE10K 2-view PSNR from 25.89 to 28.41 at roughly 0.3 M-params change (+0.5%), with a 2048 model trained from scratch performing worse than the 256 model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MVGD instead of MVSplat | novel view synthesis · RealEstate10K (2-view) | PSNR: 26.39 -> 28.41 | Helped |
| use MVGD instead of CAT3D | novel view synthesis · LLFF (3-view) | PSNR: 21.58 -> 22.39 | Helped |
| add joint depth generation to an RGB-only model (ablation) | MVGD novel view synthesis · RE10K (2-view) | PSNR: 21.46 -> 25.68 | Helped |
| add joint RGB generation to a depth-only model (ablation) | MVGD novel view synthesis · RE10K (2-view) | PSNR: —– -> 25.15 | Helped |
| add scene scale normalization (ablation) | MVGD · RE10K (2-view) | PSNR: 20.15 -> 25.89 | Helped |
| include dynamic datasets in training (ablation) | MVGD · RE10K (2-view) | PSNR: 23.24 -> 25.89 | Helped |
| use the full rather than short training schedule (ablation) | MVGD · RE10K (2-view) | PSNR: 24.44 -> 25.89 | Helped |
| increase the latent-token count from 256 to 512 (ablation) | MVGD · RE10K (2-view) | PSNR: 25.89 -> 26.33 | Helped |
| increase the latent-token count from 512 to 1024 (ablation) | MVGD · RE10K (2-view) | PSNR: 26.33 -> 27.73 | Helped |
| increase the latent-token count from 1024 to 2048 (ablation) | MVGD · RE10K (2-view) | PSNR: 27.73 -> 28.41 | Helped |
| train the 2048-latent model from scratch rather than fine-tuning incrementally (ablation) | MVGD · RE10K (2-view) | PSNR: 25.89 -> 22.09 | Hurt |
| add more conditioning views | MVGD · LLFF | Abs.Rel.: 0.105 -> 0.098 -> 0.093 | Helped |
Surprising findings
- Training a 2048-latent model from scratch for 300k steps performed worse than a 256-latent model, while incremental fine-tuning from smaller models worked better.
- Novel view synthesis was worse without scene scale normalization than without multi-task learning, indicating inaccurate geometry is more harmful than no geometry.
Limitations
- Does not model dynamic objects despite training on datasets containing them.
- Requires the target camera and conditioning cameras to be posed; relies on training diversity for zero-shot generalization.
- Higher inference cost with more latent tokens (1.1s for 2048-latent model) and stochastic outputs requiring test-time ensembling.
Experimental setup
Datasets: ArgoVerse2 BlendedMVG CO3Dv2 DROID LSD LyftL5 MVImgNet NuScenes Taskonomy HM3D Hypersim PlaceDomain
Benchmarks: RealEstate10K ACID LLFF DTU CO3D MIP-NeRF360 ScanNet SUN3D RGB-D
Metrics: PSNR SSIM LPIPS Abs.Rel. RMSE delta 1.25
Baselines: PixelNeRF GPNR AttnRend MuRF PixelSplat MVSplat FreeNeRF SimpleNeRF ZeroNVS ReconFusion CAT3D LLFF
Builds on: Recurrent Interface Networks DDIM EfficientViT-SAM Fourier positional encoding
Closest papers by meaning
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.90)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.90)
- MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation (CVPR 2024 · similarity 0.89)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.89)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.89)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.