Research map / 3D vision / 3D generation
Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiDiff, a joint 2D-3D diffusion framework that couples an SDF-space 3D diffusion model (initialized from Shap-E) with a multi-view 2D diffusion model (initialized from DeepFloyd-IF) via bidirectional guidance, enabling fast text-to-3D generation and optional refinement with ProlificDreamer.
Problem
Text-to-3D generation suffers from geometric anomalies and multi-view inconsistency when lifting 2D diffusion models (SDS optimization is slow and produces Janus artifacts), while 3D-only diffusion models trained on small 3D datasets yield poor textures.
Approach
Jointly denoise a 3D SDF diffusion model and a multi-view 2D diffusion model at each step, with 2D-to-3D guidance (back-projected multi-view features) and 3D-to-2D guidance (rendered radiance field fed as ControlNet-style residuals), plus classifier-free-style prior scales to separately control geometry and texture.
Key results
- BiDiff sampling reaches CLIP R-precision 0.79 in about 40 seconds, versus 0.83 for ProlificDreamer at 3.4 hours and 0.67 for DreamFusion at 1.1 hours.
- Using BiDiff output as initialization for ProlificDreamer gives R-precision 0.85 in about 20 minutes total.
- Ablations show removing 3D priors limits generation to training-set objects and removing 2D priors yields less realistic textures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BiDiff sampling instead of DreamFusion | Text-to-3D generation · Objaverse test prompts | CLIP R-precision: 0.67 -> 0.79 | Helped |
| Use BiDiff sampling instead of ProlificDreamer | Text-to-3D generation · Objaverse test prompts | CLIP R-precision: 0.83 -> 0.79 | Hurt |
| Use BiDiff post-optimization instead of ProlificDreamer | Text-to-3D generation · Objaverse test prompts | CLIP R-precision: 0.83 -> 0.85 | Helped |
| Remove the Shap-E conditional radiance field (ablation) | BiDiff · ShapeNet-Chair | geometry diversity | Hurt |
| Remove the pretrained 2D diffusion prior by randomly initializing its parameters (ablation) | BiDiff · ShapeNet-Chair | texture realism | Hurt |
| Remove the prior enhancement strategy (ablation) | BiDiff · ShapeNet-Chair | generation diversity | Hurt |
Limitations
- Ablations and evaluation are limited to ShapeNet-Chair and a 50-prompt Objaverse set with a single CLIP R-precision metric
- Paper notes that during training the previous-step denoised images are replaced with ground truth, creating a train-inference mismatch
Experimental setup
Datasets: ShapeNet-Chair Objaverse LVIS 40k
Benchmarks: ShapeNet-Chair Objaverse 50 test prompts
Metrics: CLIP R-precision inference time
Baselines: DreamFusion ProlificDreamer Shap-E Zero-1-to-3
Builds on: Shap-E DeepFloyd-IF ControlNet ProlificDreamer SparseNeuS
Closest papers by meaning
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.89)
- Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation (CVPR 2025 · similarity 0.89)
- Diverse and Stable 2D Diffusion Guided Text to 3D Generation with Noise Recalibration (AAAI 2024 · similarity 0.89)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.89)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.88)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.