Research map / 3D vision / 3D generation
RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RichDreamer, a joint normal-depth latent diffusion model pre-trained on LAION and fine-tuned on Objaverse, plus a depth-conditioned albedo diffusion model, to improve geometry and appearance in decoupled text-to-3D generation.
Problem
Text-to-3D pipelines that optimize surface normals with an RGB 2D diffusion model suffer from distribution mismatch between natural images and normal maps, causing unstable optimization and poor geometry; appearance also entangles shadows and specular highlights with albedo.
Approach
Learn a generalizable joint normal-depth diffusion prior (trained on LAION pseudo-labeled with monocular normal/depth estimators, then fine-tuned on Objaverse with multi-view camera conditioning) to supply SDS geometry guidance, and a depth-conditioned albedo diffusion model to regularize the diffuse material component.
Key results
- Ours (NeRF) achieves geometry CLIP score 26.0570 and appearance CLIP score 31.3551, versus MVDream's 24.8003 and 28.7331.
- Ours (Sphere) reaches appearance CLIP score 31.7099, second to ProlificDreamer's 31.8022, and geometry CLIP 25.8820.
- In user study, Ours (NeRF) received 75% of textured-model votes and 70% of geometry votes against the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Stable Diffusion guidance alongside Normal-Depth diffusion (ablation) | RichDreamer geometry generation · Text-to-3D geometry ablation | Geometry CLIP Score: 24.1070 -> 25.8820 | Helped |
| Add Stable Diffusion guidance alongside Normal-Depth diffusion (ablation) | RichDreamer geometry generation · Text-to-3D geometry ablation | Appearance CLIP Score: 29.5379 -> 31.7099 | Helped |
| Pre-train Normal-Depth diffusion on LAION before Objaverse fine-tuning (ablation) | RichDreamer geometry generation · Text-to-3D geometry ablation | Geometry CLIP Score: 24.2601 -> 25.8820 | Helped |
| Pre-train Normal-Depth diffusion on LAION before Objaverse fine-tuning (ablation) | RichDreamer geometry generation · Text-to-3D geometry ablation | Appearance CLIP Score: 29.7522 -> 31.7099 | Helped |
| Add a depth condition to albedo diffusion (ablation) | RichDreamer appearance modeling · Text-to-3D appearance ablation | Texture-geometry alignment | Helped |
| Add albedo diffusion guidance (ablation) | RichDreamer appearance modeling · Text-to-3D appearance ablation | Albedo separation from lighting | Helped |
Surprising findings
- Fine-tuning the normal-depth model only on Objaverse (without LAION pre-training) markedly degrades generalization to user prompts, showing LAION pre-training is crucial.
Limitations
- Albedo modeling is described as a preliminary approach to mixed illumination and still leaves shadows/highlights in some cases.
- Fantasia3D-style geometry optimization relies on sphere initialization or coarse geometry for stability.
- Evaluation relies mainly on CLIP scores and a user study over a limited prompt set; no public code for several baselines so comparisons used third-party implementations.
Experimental setup
Datasets: LAION-2B Objaverse Cap3D
Benchmarks: user-prompt text-to-3D set of 113 objects (geometry/appearance CLIP score) user study over 87 prompts
Metrics: CLIP score (geometry, uniform albedo rendering) CLIP score (textured appearance) user preference votes
Baselines: DreamFusion-IF Magic3D-IF TextMesh-IF ProlificDreamer Fantasia3D MVDream SweetDreamer
Builds on: Fantasia3D DreamFusion MVDream Stable Diffusion
Closest papers by meaning
- GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (CVPR 2024 · similarity 0.87)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.87)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.87)
- Wonder3D: Single Image to 3D using Cross-Domain Diffusion (CVPR 2024 · similarity 0.86)
- 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion (CVPR 2025 · similarity 0.86)
- Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.