Research map / 3D vision / 3D generation
DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamCraft3D, a hierarchical text-to-3D pipeline that lifts a 2D reference image into 3D via geometry sculpting with a 3D-aware (Zero-1-to-3) plus 2D SDS prior, then texture boosting with Bootstrapped Score Distillation using a scene-specific DreamBooth model alternately trained on the evolving renderings.
Problem
Text-to-3D and image-to-3D generation via score distillation suffers from multi-view inconsistency (Janus problem), over-smoothed or over-saturated textures, and limited realism for complex objects.
Approach
Hierarchical generation: a 2D reference image guides geometry sculpting with a hybrid 2D+3D (Zero-1-to-3) SDS prior, and texture boosting uses Bootstrapped Score Distillation, where a DreamBooth model finetuned on augmented renderings of the scene being optimized provides increasingly view-consistent guidance in an alternating optimization loop.
Key results
- On a 300-sample benchmark, DreamCraft3D reports CLIP 0.896, Contextual 1.579, PSNR 31.801, LPIPS 0.005, beating Make-it-3D and Magic123 on all four metrics.
- In a user study (32 participants, 480 responses), DreamCraft3D was preferred by 92% of users on average over the baselines.
- Bootstrapping (two rounds of alternating BSD) progressively improves texture consistency and detail relative to one-round BSD, VSD, and SDS in the ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DreamCraft3D instead of Make-it-3D or Magic123 | Image-to-3D generation · 300-sample test benchmark | CLIP: 0.872 -> 0.896; 0.843 -> 0.896 | Helped |
| Use DreamCraft3D instead of Make-it-3D or Magic123 | Image-to-3D generation · 300-sample test benchmark | PSNR: 18.937 -> 31.801; 22.838 -> 31.801 | Helped |
| Use DreamCraft3D instead of Make-it-3D or Magic123 | Image-to-3D generation · 300-sample test benchmark | Contextual: 1.609 -> 1.579; 1.628 -> 1.579 | Helped |
| Use DreamCraft3D instead of Make-it-3D or Magic123 | Image-to-3D generation · 300-sample test benchmark | LPIPS: 0.054 -> 0.005; 0.053 -> 0.005 | Helped |
| Add a viewpoint-aware 3D prior (ablation) | DreamCraft3D geometry sculpting · 3D geometry consistency | geometric consistency | Helped |
| Replace SDS with VSD for texture optimization (ablation) | DreamCraft3D texture optimization · novel-view texture | texture realism | Helped |
| Replace SDS with VSD for texture optimization (ablation) | DreamCraft3D texture optimization · multi-view texture | texture consistency | Hurt |
| Replace SDS with BSD for texture optimization (ablation) | DreamCraft3D texture optimization · novel-view texture | texture detail | Helped |
| Replace SDS with BSD for texture optimization (ablation) | DreamCraft3D texture optimization · multi-view texture | texture consistency | Helped |
| Use traditional SDS with Stable Diffusion for texture optimization (ablation) | DreamCraft3D texture optimization · novel-view texture | texture smoothness | Hurt |
| Convert NeuS to DMTet during geometry sculpting (ablation) | DreamCraft3D geometry sculpting · geometry sculpting stage | high-resolution geometry detail | Helped |
| Convert NeuS to DMTet during geometry sculpting (ablation) | DreamCraft3D geometry sculpting · geometry sculpting stage | texture quality | No clear effect |
Surprising findings
- The paper reports that using VSD with standard Stable Diffusion yields high texture inconsistency, and that Zero-1-to-3 guidance hurts texture quality, so the 3D prior is dropped during texture stage.
Limitations
- Occasionally incorporates frontal-view geometric details into texture due to depth ambiguity and depth prior inaccuracies.
- Does not explicitly separate material and lighting from the 2D reference image.
- Incorrect geometry in some failure cases (e.g., elephant nose).
Experimental setup
Datasets: multi-view renderings of the optimized 3D scene
Benchmarks: 300-image internal test benchmark (real and generated images)
Metrics: CLIP score Contextual distance PSNR LPIPS user preference
Baselines: DreamFusion Magic3D ProlificDreamer Make-it-3D Magic123
Builds on: DreamFusion ProlificDreamer Zero-1-to-3 DreamBooth NeuS
Closest papers by meaning
- Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (ICLR 2024 · similarity 0.91)
- CAD: Photorealistic 3D Generation via Adversarial Distillation (CVPR 2024 · similarity 0.90)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.90)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.90)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.89)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.