Research map / Image generation / Text-to-image diffusion
IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Refines coarse text-to-3D models (DreamFusion baseline) by generating posed multi-view images with ControlNet/Stable Diffusion image-to-image pipelines and training the 3D model with a Diffusion-GAN dual strategy using an EG3D-style discriminator.
Problem
Existing score-distillation text-to-3D methods suffer from over-saturation, inadequate detail, multi-face (Janus) artifacts and unrealistic outputs; naive use of generated multi-view images is hampered by view and content inconsistency.
Approach
Treat image-to-image-refined multi-view renderings of a coarse 3D model as real data for a randomly initialized discriminator (3D GAN style), and combine the adversarial loss with the SDS diffusion prior so that the diffusion prior removes discriminator-induced noise, speeding GAN convergence to under 1 GPU hour.
Key results
- In a user study with 45 participants (675 pairwise comparisons over 15 prompts), IT3D was preferred over the baseline 89.92% of the time on average.
- Full Diffusion-GAN refinement converges within 0.5 to 1.0 GPU hours, versus 1.5 to 2.5 GPU hours for training the baseline to 25K steps.
- Dataset generation uses about 200 posed samples, with 24 to 60 camera views and 3 to 7 samples per view.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Diffusion-GAN dual training instead of training a vanilla 3D GAN to full convergence | IT3D refinement · Text-to-3D refinement | training time: 0.5 to 1.0 GPU hours vs over 10.0 GPU hours | Helped |
Surprising findings
- Plain L2 loss on the inconsistent generated dataset pulls the 3D model toward the average content, so a discriminator is needed to exploit the high-variance data.
- The method tolerates failure cases in the generated dataset (e.g., misaligned views) and can still recover coarse models with mild geometry or Janus problems.
Limitations
- Performance is bounded by the image-to-image pipeline; it fails when the input prompt is too complicated.
- Mini-dataset (~200 samples) is small, so the discriminator alone overfits and needs the diffusion prior.
- Experiments use one fixed dataset per model without the Dataset Update strategy; evaluation is mainly a user study with 15 prompts.
Experimental setup
Datasets: Synthetic multi-view dataset generated by ControlNet + Stable Diffusion image-to-image from coarse NeRF renderings
Benchmarks: User study over 15 prompts
Metrics: user preference rate (pairwise comparisons)
Baselines: DreamFusion (Stable DreamFusion) DreamBooth3D
Builds on: DreamFusion EG3D ControlNet Stable Diffusion Perp-Neg
Closest papers by meaning
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.91)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.91)
- IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation (ICML 2024 · similarity 0.91)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.90)
- DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors (CVPR 2024 · similarity 0.90)
- Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.