Research map / 3D vision / 3D generation
MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates mutually consistent multi-view RGB-D images from a single RGB input using a diffusion model with depth-projection-based 3D cross-attention, adapted from Zero-1-to-3.
Problem
Single-view 3D inference currently relies on novel-view diffusion models whose generated views are not mutually 3D-consistent, requiring costly score-distillation to recover 3D geometry.
Approach
Jointly denoise multiple RGB-D views and use the noisy intermediate depth estimates to reproject and sample features across views (depth-guided 3D cross-attention with depth-biased ray sampling), so multi-view consistency is enforced and geometry is read directly by unprojecting predicted depth maps.
Key results
- On Objaverse novel view synthesis, MVD-Fusion reaches 21.19 PSNR / 0.835 SSIM / 0.146 LPIPS versus 19.22 PSNR for SyncDreamer and 17.37 for Zero123.
- On GSO 3D reconstruction, MVD-Fusion achieves Chamfer distance 0.031, best among direct methods (One-2-3-45 0.063, Point-E 0.043, Shap-E 0.044) and better than most optimization-based methods, though SyncDreamer obtains 0.026.
- On CO3D (51 categories), MVD-Fusion gets 0.701 SSIM and 0.220 LPIPS versus PixelNeRF's 0.484 and 0.378, while PixelNeRF has slightly higher PSNR (17.64 vs 17.16).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use depth-guided multi-view generation instead of Zero123 | MVD-Fusion · Objaverse test | PSNR: 17.37 -> 21.19 | Helped |
| use depth-guided multi-view generation instead of SyncDreamer | MVD-Fusion · Objaverse test | LPIPS: 0.176 -> 0.146 | Helped |
| use depth-guided multi-view generation instead of SyncDreamer | MVD-Fusion · GSO | SSIM: 0.796 -> 0.790 | Hurt |
| use depth-guided multi-view generation instead of SyncDreamer | MVD-Fusion · GSO | PSNR: 18.95 -> 19.53 | Helped |
| use depth-guided multi-view generation instead of PixelNeRF | MVD-Fusion · CO3D 51 categories | PSNR: 17.64 -> 17.16 | Hurt |
| use depth-guided multi-view generation instead of PixelNeRF | MVD-Fusion · CO3D 51 categories | LPIPS: 0.378 -> 0.220 | Helped |
| directly unproject predicted depth maps rather than use a direct 3D baseline | MVD-Fusion · GSO | Chamfer Distance: 0.044 -> 0.031 | Helped |
Surprising findings
- On GSO, MVD-Fusion's SSIM (0.790) is slightly below SyncDreamer (0.796) despite better PSNR and LPIPS.
- PixelNeRF achieves slightly higher PSNR than MVD-Fusion on CO3D, which the authors attribute to blurry mean predictions being optimal under uncertainty.
Limitations
- Multi-view consistency is encouraged through network inductive biases but not guaranteed, so slight inconsistencies can occur.
- Coarse depth maps do not capture fine geometric details; optimization may be needed to refine them.
- Trained on clean unoccluded objects, so it is not directly applicable to cluttered scenes with partially visible objects.
Experimental setup
Datasets: Objaverse CO3D Google Scanned Objects
Benchmarks: Objaverse held-out test set Google Scanned Objects CO3D
Metrics: PSNR SSIM LPIPS Chamfer distance user preference win rate
Baselines: Zero-1-to-3 SyncDreamer PixelNeRF RealFusion Magic123 One-2-3-45 Point-E Shap-E
Builds on: Zero-1-to-3 Stable Diffusion SyncDreamer DDPM
Closest papers by meaning
- Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion (CVPR 2025 · similarity 0.90)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.90)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.90)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.89)
- HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D (CVPR 2024 · similarity 0.88)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.