Research map / Image generation / Text-to-image diffusion
Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Stable Diffusion into a four-step native Triplane text-to-mesh generator (TriplaneTurbo) trained without 3D ground truth, by distilling multi-view diffusion teachers through progressive rendering distillation and a LoRA-based parameter-efficient adaptation.
Problem
Text-to-3D feed-forward generators are limited by the scarcity and poor quality of 3D training data, so they produce low-quality meshes and generalize poorly to complex prompts.
Approach
Progressive Rendering Distillation denoises latents from random noise over K=4 SD U-Net steps, decodes each step's latent into a 3D Triplane, and supervises renderings with multi-view teachers (SD, MVDream, RichDreamer) via score distillation, removing the need for 3D ground truth. PETA adds plane-specific LoRAs (about 2.5% extra parameters) to adapt the frozen SD for six-plane Triplane output.
Key results
- TriplaneTurbo reaches CLIP score 68.2 and R@1 32.3 with 1.23 s latency, outperforming LGM (67.4, 28.3, 56.1 s) and PI3D (65.9, 25.2, 3.00 s).
- Scaling training text from 360K to 1.6M prompts raises CLIP score to 75.1 and R@1 to 46.0.
- PETA adds 2.5% trainable parameters; full-parameter tuning drops CLIP score to 35.8 versus 68.2 with PETA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PRD with K=4 instead of K=1 denoising step (ablation) | TriplaneTurbo · DreamFusion gallery test prompts | CLIP Score: 50.9 -> 68.2 | Helped |
| Use PRD with K=4 instead of K=1 denoising step (ablation) | TriplaneTurbo · DreamFusion gallery test prompts | R@1: 14.4 -> 32.3 | Helped |
| Increase PRD denoising steps from K=2 to K=4 (ablation) | TriplaneTurbo · DreamFusion gallery test prompts | CLIP Score: 62.6 -> 68.2 | Helped |
| Increase PRD denoising steps from K=2 to K=4 (ablation) | TriplaneTurbo · DreamFusion gallery test prompts | R@1: 22.4 -> 32.3 | Helped |
| Use PETA instead of full-parameter fine-tuning (ablation) | SD v2.1-base adapted for TriplaneTurbo · DreamFusion gallery test prompts | CLIP Score: 35.8 -> 68.2 | Helped |
| Use PETA instead of full-parameter fine-tuning (ablation) | SD v2.1-base adapted for TriplaneTurbo · DreamFusion gallery test prompts | R@1: 0.35 -> 32.3 | Helped |
| Use plane-specific PETA LoRAs instead of vanilla LoRA (ablation) | SD v2.1-base adapted for TriplaneTurbo · DreamFusion gallery test prompts | CLIP Score: 54.2 -> 68.2 | Helped |
| Use plane-specific PETA LoRAs instead of vanilla LoRA (ablation) | SD v2.1-base adapted for TriplaneTurbo · DreamFusion gallery test prompts | R@1: 11.1 -> 32.3 | Helped |
| Scale training text prompts from 360K to 1.6M | TriplaneTurbo · DreamFusion gallery test prompts | CLIP Score: 68.2 -> 75.1 | Helped |
| Scale training text prompts from 360K to 1.6M | TriplaneTurbo · DreamFusion gallery test prompts | R@1: 32.3 -> 46.0 | Helped |
| Use TriplaneTurbo instead of the strongest baseline LGM | Text-to-mesh generation · DreamFusion gallery test prompts | CLIP Score: 67.4 -> 68.2 | Helped |
| Use TriplaneTurbo instead of the strongest baseline LGM | Text-to-mesh generation · DreamFusion gallery test prompts | Latency (/sec): 56.1 -> 1.23 | Helped |
Surprising findings
- Single-step (K=1) distillation fails badly (CLIP 50.9), while K=2 and K=4 progressively recover quality.
Limitations
- Precise counts of multiple objects in a prompt are hard to generate.
- Full-body humans show limited facial and hand detail due to Triplane representation.
Experimental setup
Datasets: Cap3D captions (360K prompts) 1.6M collected text prompts DreamFusion gallery prompts (evaluation)
Benchmarks: DreamFusion gallery prompts
Metrics: CLIP score CLIP R-precision latency (s)
Baselines: Shape-E Direct3D 3DTopia PI3D GVGEN LN3Diff LGM
Builds on: Stable Diffusion MVDream RichDreamer Asynchronous Score Distillation LoRA
Closest papers by meaning
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.91)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.90)
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.90)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.90)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.90)
- Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.