Research map / 3D vision / 3D generation
Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sculpt3D improves multi-view consistency of text-to-3D generation by retrieving a 3D reference shape with OpenShape, supervising sparse keypoint rays for geometry, and modulating 2D diffusion outputs with a T2I-Adapter from template views, without retraining the diffusion model.
Problem
Text-to-3D generation distilled from 2D diffusion models produces inconsistent appearances (e.g., faces on back views) and inaccurate shapes (e.g., extra legs), and prior fixes retrain diffusion models on rendered 3D data at the cost of generation quality.
Approach
Inject 3D priors from a retrieved reference 3D object without retraining the 2D model: sparse ray sampling supervises only keypoints of the template shape (letting diffusion prune/grow points), and a image adapter converts template views into the generated style to correct view-specific appearance via the diffusion prior.
Key results
- Ours achieves quality 53.6 and alignment 49.3 on T3Bench, versus 51.1 and 47.8 for ProlificDreamer.
- 3D consistent rate of 76% on 50 prompts, versus 32% for ProlificDreamer and 34% for DreamFusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add retrieved-template shape and appearance priors with sparse geometry supervision and appearance modulation | ProlificDreamer · T3Bench | Quality: 51.1 -> 53.6 | Helped |
| add retrieved-template shape and appearance priors with sparse geometry supervision and appearance modulation | ProlificDreamer · T3Bench | Alignment: 47.8 -> 49.3 | Helped |
| add retrieved-template shape and appearance priors with sparse geometry supervision and appearance modulation | ProlificDreamer · T3Bench | Cons. Rate: 32% -> 76% | Helped |
| replace sparse keypoint shape supervision with full shape supervision (ablation) | Sculpt3D · Text-to-3D generation | shape diversity | Hurt |
| increase sparse selection factor to 16 (ablation) | Sculpt3D · Text-to-3D generation | shape variation from the reference | Helped |
| remove appearance modulation (ablation) | Sculpt3D · Text-to-3D generation | appearance consistency | Hurt |
| add appearance modulation using the template’s view-specific patterns (ablation) | Sculpt3D · Text-to-3D generation | appearance consistency | Helped |
| use generated rough shapes to re-retrieve matching reference shapes (ablation) | Sculpt3D · Text-to-3D generation with sparse selection factor 16 | reference-shape matching | Helped |
| apply explicit geometry supervision when the retrieved shape exceeds the 3D dataset prior | Sculpt3D · Text-to-3D generation | generation success | Hurt |
Surprising findings
- Fully supervising NeRF density to a template makes results converge to the reference shape and reduces diversity, and can cause incorrect appearances such as multiple dog faces.
Limitations
- Fails when the initial retrieved shape lies outside the 3D dataset's prior, since geometry is explicitly constrained.
- Generating an unconstrained initial shape for re-retrieval often fails due to limitations of existing generation methods.
- Appearance modulation is optional and applied only in hard cases.
Experimental setup
Datasets: T3Bench prompts external 3D database (retrieval)
Benchmarks: T3Bench
Metrics: quality score (multi-view CLIP, T3Bench) alignment score (GPT-4, T3Bench) 3D consistent rate (manual count)
Baselines: DreamFusion Latent-NeRF Magic3D Fantasia3D ProlificDreamer
Builds on: ProlificDreamer DreamFusion Latent-NeRF threestudio T2I-Adapter
Closest papers by meaning
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.90)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.90)
- Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion (CVPR 2025 · similarity 0.90)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.90)
- Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.