Research map / 3D vision / 3D generation
Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sherpa3D builds a coarse 3D prior with Shap-E and uses its normal maps and CLIP features as structural and semantic guidance during SDS-based 2D lifting with Stable Diffusion to produce high-fidelity, multi-view consistent text-to-3D assets.
Problem
Text-to-3D methods based on 2D lifting (SDS) suffer from view-agnostic ambiguity and multi-face Janus problems, while 3D diffusion models produce low-quality, poorly generalizable assets due to scarce 3D data.
Approach
Use a coarse 3D prior from a pretrained 3D diffusion model as guidance during 2D SDS lifting: a structural loss matching Gaussian-smoothed gradient maps of rendered normals, and a semantic loss matching CLIP features of normal maps across views, with an annealed weight on the guidance.
Key results
- CLIP L/14 R-Precision of 79.3% for Sherpa3D vs 76.1% for Magic3D and 75.0% for DreamFusion.
- User study (50 volunteers) multi-view consistency 8.95 vs 7.02 best baseline for overall quality; Sherpa3D overall quality 8.74 vs ProlificDreamer 7.02.
- Generation takes about 25 minutes on a single RTX 3090 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Sherpa3D's coarse 3D prior with structural and semantic guidance for 2D lifting | Text-to-3D generation · Object-centric COCO | CLIP R-Precision (CLIP B/32): 71.5 -> 72.3 | Helped |
| Use Sherpa3D's coarse 3D prior with structural and semantic guidance for 2D lifting | Text-to-3D generation · Object-centric COCO | CLIP R-Precision (CLIP B/16): 73.8 -> 75.6 | Helped |
| Use Sherpa3D's coarse 3D prior with structural and semantic guidance for 2D lifting | Text-to-3D generation · Object-centric COCO | CLIP R-Precision (CLIP L/14): 76.1 -> 79.3 | Helped |
| Use coarse 3D prior guidance in Sherpa3D | Text-to-3D generation · User study | Multi-view consistency: 6.09 -> 8.95 | Helped |
| Use coarse 3D prior guidance in Sherpa3D | Text-to-3D generation · User study | Overall generation quality: 7.02 -> 8.74 | Helped |
| Add structural guidance from the coarse 3D prior (ablation) | Sherpa3D · A head of the Terracotta Army | Geometric fidelity | Helped |
| Add semantic guidance from the coarse 3D prior (ablation) | Sherpa3D · A head of the Terracotta Army | Multi-view consistency | Helped |
| Add step annealing to balance the influence of 3D guidance (ablation) | Sherpa3D · A head of the Terracotta Army | Generation quality | Helped |
Surprising findings
- Zero123 fine-tuned on 3D data overfits and degrades on open-vocabulary prompts, per the authors' comparison.
Limitations
- Output quality limited by backbones Shap-E and Stable Diffusion v2.1 base.
- Relies on the quality of the coarse 3D prior from the 3D diffusion model.
Experimental setup
Datasets: COCO captions (100 prompts for evaluation)
Benchmarks: COCO object-centric prompts (100 prompts)
Metrics: CLIP R-Precision user study multi-view consistency score user study overall quality score
Baselines: Shap-E DreamFusion Magic3D Fantasia3D ProlificDreamer Zero123
Builds on: DreamFusion Fantasia3D DMTet Shap-E
Closest papers by meaning
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.84)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.84)
- TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields (ICLR 2024 · similarity 0.84)
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.84)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.84)
- Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.