Research map / Image generation / Text-to-image diffusion
PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion v2.1 to generate triplanes treated as sets of pseudo-RGB images from text, mixed with LAION real images, then refines samples with SDS in about 3 minutes.
Problem
Text-to-3D generation is limited by the scarcity of large, high-quality 3D datasets, so 3D diffusion models generalize poorly and optimization-based SDS methods are slow and inconsistent.
Approach
Represent a 3D shape as a triplane split into six pseudo-RGB images, fine-tune a pretrained text-to-image diffusion model (with global attention across pseudo-images) to generate them, co-train with real images to keep 2D generalization, and refine the sample with 2D SDS.
Key results
- PI3D reaches CLIP score 65.9 with ViT-B-32 versus 62.0 for Shap·E and 22.3 for Point·E, and R-Precision@1 of 25.2 versus 16.0 for Shap·E.
- Pseudo-image sampling takes about 3 seconds per triplane, and the SDS refinement takes about 3 minutes on an A100.
- Training with no real images drops ViT-B-32 CLIP score to 61.5 and R-Precision@1 to 15.7, versus 65.9 and 25.2 with 50% real images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PI3D instead of Shap·E | text-to-3D generation · DreamFusion gallery prompts | CLIP Score (ViT-B-32): 62.0 -> 65.9 | Helped |
| Use PI3D instead of Shap·E | text-to-3D generation · DreamFusion gallery prompts | R-Precision@1 (ViT-B-32): 16.0 -> 25.2 | Helped |
| Use PI3D instead of Shap·E | text-to-3D generation · DreamFusion gallery prompts | R-Precision@10 (ViT-L-14): 43.7 -> 63.3 | Helped |
| Add real images to pseudo-image diffusion training (ablation) | PI3D · DreamFusion gallery prompts | R-Precision@1 (ViT-B-32): 15.7 -> 25.2 | Helped |
| Add real images to pseudo-image diffusion training (ablation) | PI3D · DreamFusion gallery prompts | R-Precision@1 (ViT-L-14): 16.7 -> 29.3 | Helped |
| Set classifier-free guidance scale to 5.0 rather than 1.0 (ablation) | PI3D · DreamFusion gallery prompts | R-Precision@1 (ViT-B-32): 8.4 -> 25.2 | Helped |
| Set classifier-free guidance scale to 5.0 rather than 1.0 (ablation) | PI3D · DreamFusion gallery prompts | R-Precision@1 (ViT-L-14): 10.1 -> 29.3 | Helped |
| Add depth loss during triplane fitting (ablation) | PI3D triplane representation · triplane fitting | geometry plausibility | Helped |
| Increase classifier-free guidance scale too far (ablation) | PI3D · text-to-3D generation | geometry quality | Hurt |
Surprising findings
- Mixing in 2D real images during 3D diffusion training substantially improves generalization to complex prompts, even though the 3D data is small.
Limitations
- Per-object triplane fitting makes training cost grow linearly with dataset size.
- Triplane feature dimension limits representational capacity for detailed 3D generation.
- Scaling to higher-dimensional features would add pseudo-images and complicate cross-plane modeling.
Experimental setup
Datasets: Objaverse LVIS subset Cap3D captions LAION-5B
Benchmarks: DreamFusion gallery prompts (415)
Metrics: CLIP score CLIP R-Precision@1 CLIP R-Precision@10
Baselines: Point·E Shap·E DreamFusion Fantasia3D ProlificDreamer MVDream ATT3D
Builds on: Stable Diffusion Score Distillation Sampling Shap·E Cap3D threestudio
Closest papers by meaning
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.91)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.90)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.90)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.89)
- Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion (CVPR 2024 · similarity 0.89)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.