Research map / 3D vision / 3D generation
MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MVGenMaster, a multi-view diffusion model conditioned on metric-depth-warped RGB and canonical coordinate map 3D priors, trained on a 1.6M-scene dataset (MvD-1M), with a training-free key-rescaling trick to generate up to 100 novel views in one forward pass.
Problem
Versatile novel view synthesis from arbitrary reference and target views is hard: existing diffusion NVS methods lack 3D priors (poor out-of-domain consistency), depend on iterative anchor-based generation, dataset updates or test-time optimization, and degrade when generating many views.
Approach
Warp RGB pixels and canonical coordinate maps from reference views to targets using metric depth (aligned monocular depth to SfM via RANSAC) and camera poses, feed them as 3D priors into an SD2-based multi-view diffusion model with full cross-view attention, and scale view count at inference by multiplying reference keys by a constant (key-rescaling).
Key results
- On the zero-shot benchmark with 25 total views, 3-view MVGenMaster reaches 13.718 PSNR vs 10.865 for CAT3D* and 11.431 for ViewCrafter.
- Key-rescaling with 100 views at once improves MipNeRF-360 PSNR from 14.751 (no rescaling) to 15.640; anchor-based 28/100 with rescaling gives 13.879 without it.
- On 3DGS reconstruction from 3 views (100 frames), MVGenMaster reaches 14.669 PSNR on Co3Dv2-style comparison set versus 11.758 for CAT3D* on Tanks-and-Temples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add metric-depth-warped 3D priors | MVGenMaster · CO3D+MVImgNet unordered | PSNR: 17.540 -> 21.466 | Helped |
| Add metric-depth-warped 3D priors | MVGenMaster · DL3DV+Real10k unordered | PSNR: 14.006 -> 18.296 | Helped |
| Add metric-depth-warped 3D priors | MVGenMaster · Zero-shot datasets unordered | PSNR: 10.878 -> 15.533 | Helped |
| Add qk-norm before self-attention (ablation) | MVGenMaster · In-domain validation set | PSNR: 15.348 -> 15.521 | Helped |
| Add CCM prior fused by convolution (ablation) | MVGenMaster · In-domain validation set | PSNR: 15.521 -> 17.521 | Helped |
| Inject CCM prior through cross-attention instead of convolutional input fusion (ablation) | MVGenMaster · In-domain validation set | PSNR: 17.521 -> 15.830 | Hurt |
| Use warped RGB pixels rather than warped latents (ablation) | MVGenMaster · In-domain validation set | PSNR: 17.514 -> 17.651 | Helped |
| Add domain switcher (ablation) | MVGenMaster · Zero-shot test set | PSNR: 14.286 -> 14.956 | Helped |
| Add multi-scale training (ablation) | MVGenMaster · Zero-shot test set | PSNR: 14.956 -> 15.211 | Helped |
| Add EMA (ablation) | MVGenMaster · Zero-shot test set | PSNR: 15.211 -> 15.641 | Helped |
| Add key-rescaling (ablation) | MVGenMaster · MipNeRF-360, 100 target views | PSNR: 14.833 -> 15.640 | Helped |
Surprising findings
- Naively extending views beyond about 25 causes large quality degradation, attributed to attention dilution.
- Injecting CCM features via cross-attention performed worse than simply adding them via convolutional encoding.
Limitations
- Attention dilution and degradation when total views exceed about 28 without key-rescaling; key-rescaling gamma=1.2 is chosen empirically.
- Zero-shot performance is still modest in absolute PSNR (around 13-16 dB), and ViewCrafter edges out CAT3D* on the zero-shot 1-view setting due to Dust3R point clouds.
- Relies on accurate metric depth and camera poses; Dust3R struggles with complex scenes and SfM-based alignment is needed for training.
Experimental setup
Datasets: Co3Dv2 MVImgNet DL3DV GL3D Scannet++ 3D-Front Real10k ACID Objaverse Megascenes Aerial Streetview
Benchmarks: Co3Dv2 MVImgNet DL3DV Real10k DTU MipNeRF-360 Tanks-and-Temples ETH3D
Metrics: PSNR SSIM LPIPS
Baselines: ViewCrafter CAT3D MVSplat
Builds on: CAT3D Stable Diffusion 2 Wonder3D ViewCrafter Depth-Pro
Closest papers by meaning
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.91)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation (CVPR 2024 · similarity 0.90)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.89)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.89)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.