Research map / 3D vision / 3D generation
Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds 3D awareness to SDS-based text-to-3D by conditioning Stable Diffusion on sparse projected depth maps from a point cloud (Point-E or MCC) via a ControlNet-style injector, plus semantic code sampling and LoRA.
Problem
SDS-based text-to-3D generation produces inconsistent, distorted geometry (e.g., the Janus problem) because the 2D diffusion model lacks camera-pose and 3D awareness.
Approach
Condition a frozen pretrained 2D diffusion model on view-aligned sparse depth maps projected from a coarse 3D point cloud generated from a text-derived image, training only a sparse depth injector (ControlNet-style); semantic code (generated image plus optimized text embedding) and LoRA keep semantics consistent across views.
Key results
- Combining 3DFuse with SJC substantially outperforms baseline SJC on the proposed COLMAP-based 3D consistency variance metric over 42 scenes.
- In a 102-participant user study, 3DFuse-SJC is preferred by most participants for 3D coherence, prompt adherence, and overall quality over Stable-DreamFusion and SJC.
- The method is shown to be adaptable to DreamFusion, SJC, and ProlificDreamer baselines with qualitative improvements in geometric consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3D consistency injection using projected sparse depth maps | SJC · 42 generated 3D scenes | COLMAP-based 3D consistency variance | Helped |
| Add semantic code sampling (ablation) | 3DFuse · Text-to-3D generation across viewpoints | semantic and geometric consistency | Helped |
| Add the sparse depth injector (ablation) | 3DFuse with optimized embedding and LoRA · Text-to-3D generation | geometric consistency | Helped |
Surprising findings
- A CLIP-based metric can give higher scores to geometrically inconsistent scenes with multiple frontal faces, so it is ineffective for measuring 3D consistency.
Limitations
- Inherits the pretrained diffusion model's difficulty in faithfully following complex user prompts.
- May reflect societal biases present in the datasets and text-to-image models.
- Coarse point cloud from off-the-shelf models contains errors and artifacts that the injector must handle.
Experimental setup
Datasets: Co3D MiDaS-predicted text-to-image depth pairs
Benchmarks: 42 generated 3D scenes with COLMAP-based consistency metric user study with 102 participants
Metrics: COLMAP camera pose variance (3D consistency) user preference (3D coherence, prompt adherence, overall qua CLIP-based metric (critiqued)
Baselines: DreamFusion (Stable-DreamFusion) SJC ProlificDreamer Zero-1-to-3 Magic-123
Builds on: DreamFusion Score Jacobian Chaining ProlificDreamer ControlNet Point-E
Closest papers by meaning
- Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior (CVPR 2024 · similarity 0.91)
- Retrieval-Augmented Score Distillation for Text-to-3D Generation (ICML 2024 · similarity 0.91)
- Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data (CVPR 2025 · similarity 0.91)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.91)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.91)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.