Research map / 3D vision / 3D generation
MVDream: Multi-view Diffusion for 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion 2.1 into a multi-view diffusion model with inflated 3D self-attention and camera embeddings, then uses it as a 3D prior for score distillation sampling and multi-view DreamBooth.
Problem
2D-lifting text-to-3D methods using SDS suffer from multi-face (Janus) artifacts and content drift across views because 2D diffusion priors lack multi-view and 3D awareness.
Approach
Train a diffusion model that jointly generates a set of consistent multi-view images by inflating the pretrained 2D self-attention into 3D (all views' tokens attend to each other), adding camera embeddings, and co-training with 2D LAION data to retain generalizability; use it as a multi-view SDS prior with an x0-reconstruction loss and CFG rescale.
Key results
- Multi-view diffusion trained with 3D + LAION data reaches FID 39.04, IS 12.97, CLIP 30.38 on the validation set, close to the validation-set IS 12.90 and CLIP 30.12.
- In a user study over 40 prompts (914 feedbacks from 38 users), 78% of users preferred MVDream's 3D model over the best of the baselines on average.
- The 4-view model barely shows multi-face/consistency issues, while the 1-view model still has severe Janus problems in SDS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LAION 2D data during training (ablation) | MVDream · Multi-view image generation | FID: 40.38 -> 39.04 | Helped |
| Add LAION 2D data during training (ablation) | MVDream · Multi-view image generation | IS: 12.33\pm0.63 -> 12.97\pm0.60 | Helped |
| Add LAION 2D data during training (ablation) | MVDream · Multi-view image generation | CLIP score: 29.69\pm3.36 -> 30.38\pm3.50 | Helped |
| Replace temporal self-attention with inflated 2D self-attention for 3D cross-view attention (ablation) | MVDream · Multi-view image generation (8 views, 90-degree change) | multi-view consistency | Helped |
| Add a new 3D self-attention module rather than inflating existing 2D attention (ablation) | MVDream · Multi-view image generation (8 views, 90-degree change) | generation quality | Hurt |
| Increase the diffusion model's training views from 1 to 2 (ablation) | MVDream · Text-to-3D generation | multi-view consistency | Helped |
| Increase the diffusion model's training views from 2 to 4 (ablation) | MVDream · Text-to-3D generation | multi-view consistency | Helped |
| Add time-step annealing during score distillation (ablation) | MVDream SDS · Text-to-3D generation | shape completeness | Helped |
| Add negative prompts during score distillation (ablation) | MVDream SDS · Text-to-3D generation | visual style | Helped |
| Add CFG rescale during score distillation (ablation) | MVDream SDS · Text-to-3D generation | texture color naturalness | Helped |
| Use x0-reconstruction loss instead of original SDS (ablation) | MVDream SDS · Text-to-3D generation | SDS performance | No clear effect |
| Use CFG rescale with x0-reconstruction loss (ablation) | MVDream SDS · Text-to-3D generation | color saturation | Helped |
Surprising findings
- Adding a new 3D self-attention layer (rather than inflating existing 2D attention) severely degraded quality without learning consistency.
- Fine-tuning with 3D data alone deteriorated image quality, and mixing in 2D LAION data mitigated this.
Limitations
- Current model generates only 256x256 images, lower than the 512x512 of the base Stable Diffusion.
- Generalizability is limited by the base model, and generated styles (lighting, texture) are affected by the rendered Objaverse dataset.
- Trained only on 4 orthogonal views at the same elevation, with multi-view SDS limited to that setup.
Experimental setup
Datasets: Objaverse LAION
Benchmarks: Objaverse validation set (FID, IS, CLIP score) 40 user-collected text-to-3D prompts
Metrics: FID IS CLIP score user preference
Baselines: dreamfusion-IF magic3d-IF text2mesh-IF prolificdreamer DreamBooth3D Zero123-XL
Builds on: Stable Diffusion DreamFusion DreamBooth Video diffusion inflated attention threestudio
Closest papers by meaning
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.91)
- Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion (CVPR 2024 · similarity 0.91)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.90)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.90)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.90)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.