Research map / Image generation / Text-to-image diffusion
EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion 2.1 on Objaverse renders tiled as four orthogonal views, then uses this orthogonal-view diffusion prior fused with the standard 2D prior in a dynamic SDS schedule to generate text-to-3D models that avoid the Janus problem.
Problem
Text-to-3D generation with diffusion-based score distillation suffers from the Janus problem (multi-faced 3D models) because 2D diffusion models fail to follow view instructions in text prompts.
Approach
Fine-tune a 2D diffusion model to generate a 2x2 composite of four mutually consistent orthogonal-view images, use it as a geometric prior for SDS on NeuS/DMTet representations, and blend it with the pre-trained 2D diffusion prior via a weight that shifts over optimization.
Key results
- Ours achieves CLIP 30.33 and FID 284.98, beating DreamFusion (28.40/374.44), Magic3D (29.15/310.57) and TextMesh (27.65/305.77).
- In a user study with 50 participants, 84.27% preferred the proposed method's results over competitors, with a rating of 3.74 versus 1.73-2.44 for baselines.
- Ablation: removing the orthogonal-view prior gives FID 372.86 and removing the pre-trained prior gives FID 369.73, versus 342.62 for the full fusion network.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EfficientDreamer instead of Magic3D | Text-to-3D generation · Text-to-3D evaluation | CLIP score: 29.15 -> 30.33 | Helped |
| use EfficientDreamer instead of TextMesh | Text-to-3D generation · Text-to-3D evaluation | FID: 305.77 -> 284.98 | Helped |
| add the orthogonal-view diffusion prior (ablation) | 3D synthesis fusion network · Text-to-3D evaluation | CLIP score: 31.13 -> 32.55 | Helped |
| add the orthogonal-view diffusion prior (ablation) | 3D synthesis fusion network · Text-to-3D evaluation | FID: 372.86 -> 342.62 | Helped |
| add the pre-trained diffusion prior (ablation) | 3D synthesis fusion network · Text-to-3D evaluation | CLIP score: 32.05 -> 32.55 | Helped |
| add the pre-trained diffusion prior (ablation) | 3D synthesis fusion network · Text-to-3D evaluation | FID: 369.73 -> 342.62 | Helped |
| increase supervision from two views to four views (ablation) | EfficientDreamer · Text-to-3D evaluation | CLIP score: 30.92 -> 31.47 | Helped |
| increase supervision from two views to four views (ablation) | EfficientDreamer · Text-to-3D evaluation | FID: 399.58 -> 345.72 | Helped |
| increase supervision from three views to four views (ablation) | EfficientDreamer · Text-to-3D evaluation | CLIP score: 30.48 -> 31.47 | Helped |
| increase supervision from three views to four views (ablation) | EfficientDreamer · Text-to-3D evaluation | FID: 373.04 -> 345.72 | Helped |
Limitations
- Orthogonal-view prior relies on a 3D dataset (Objaverse) that is small and has simple topology and texture, which can cause local artifacts.
- Two- and three-view supervision still show multi-face artifacts, so four views are needed.
- Comparison metrics (CLIP, FID) are computed on a small set of prompts and the 3D baselines are re-implemented in a unified framework.
Experimental setup
Datasets: Objaverse Cap3D
Benchmarks: CLIP score and FID on generated 3D models (prompt set not specified) user study on 22 prompts
Metrics: CLIP score FID user rating score (1-4) user preference (%)
Baselines: DreamFusion Magic3D TextMesh Perp-Neg
Builds on: DreamFusion Magic3D ProlificDreamer Perp-Neg NeuS
Closest papers by meaning
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.91)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.91)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.90)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.90)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.