Research map / 3D vision / 3D generation
DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DMV3D, a single-stage multi-view diffusion model whose denoiser is an LRM-style transformer that reconstructs a triplane NeRF from noisy views and renders denoised images, trained only with image losses for text- and image-conditioned 3D generation.
Problem
Generic, category-agnostic 3D generation from text or a single image is slow with SDS optimization and hard for 3D diffusion models that need 3D supervision or suffer from two-stage latent-space issues.
Approach
Use a large transformer sparse-view reconstruction model (LRM-style triplane NeRF) as the denoiser inside a 2D multi-view diffusion process, supervising with rendering losses at input and novel views, so 3D generation happens within the 2D denoising loop without 3D assets; camera conditioning uses Plucker rays and time uses adaLN-Zero.
Key results
- On ABO single-image reconstruction, DMV3D reaches FID 27.88 and PSNR 24.15 versus 31.59 and 17.33 for Zero-1-to-3.
- On text-to-3D with ViT-L/14, DMV3D gets R-Precision 55.14 versus 51.40 for Shap-E.
- Full model takes about 30 seconds per asset on a single A100 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use four rather than one input view (ablation) | DMV3D-S · GSO | FID: 70.59 -> 35.16 | Helped |
| Use four rather than two input views (ablation) | DMV3D-S · GSO | FID: 47.69 -> 35.16 | Helped |
| Use six rather than four input views (ablation) | DMV3D-S · GSO | FID: 35.16 -> 39.11 | No clear effect |
| Add novel-view rendering supervision (ablation) | DMV3D-S · GSO | FID: 102.00 -> 35.16 | Helped |
| Use Plucker-coordinate camera conditioning instead of adaLN-Zero camera conditioning (ablation) | DMV3D-S · GSO | FID: 43.31 -> 35.16 | Helped |
| Add MVImgNet training data (ablation) | DMV3D · GSO | PSNR: 21.85 -> 22.57 | Mixed |
| Add MVImgNet training data (ablation) | DMV3D · GSO | FID: 27.76 -> 30.01 | Hurt |
| Add MVImgNet training data (ablation) | DMV3D · GSO | CLIP: 0.924 -> 0.928 | Helped |
| Add MVImgNet training data (ablation) | DMV3D · GSO | LPIPS: 0.128 -> 0.126 | Helped |
| Add MVImgNet training data (ablation) | DMV3D · GSO | CD: 0.0378 -> 0.0395 | Hurt |
| Use DMV3D rather than Magic123 for single-image reconstruction | DMV3D · ABO | PSNR: 18.47 -> 24.15 | Helped |
| Use DMV3D rather than Magic123 for single-image reconstruction | DMV3D · GSO | LPIPS: 0.159 -> 0.126 | Helped |
Surprising findings
- Supervising only the input-view denoised outputs leads to degenerate solutions where inputs are pasted on view-aligned planes; novel-view supervision is critical (GSO FID rises from 35.16 to 102.00 without it).
- Adding MVImgNet improves generalization but slightly reduces some quantitative scores in ablation, e.g. FID 27.76 without versus 30.01 with.
Limitations
- Generated textures for unseen parts lack high-frequency detail and have washed-out colors.
- Input images and triplanes are low resolution.
- Only supports object images without background; trained from scratch without 2D foundation model priors.
Experimental setup
Datasets: Objaverse MVImgNet Cap3D captions
Benchmarks: GSO ABO Shap-E text prompts
Metrics: FID CLIP similarity PSNR LPIPS Chamfer distance SSIM CLIP R-Precision average precision
Baselines: Point-E Shap-E Zero-1-to-3 One-2-3-45 Magic123
Builds on: LRM RenderDiffusion DiT DDIM Stable Diffusion
Closest papers by meaning
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.91)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.89)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.88)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.88)
- ViewFusion: Towards Multi-View Consistency via Interpolated Denoising (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.