Research map / Image generation / Text-to-image diffusion
DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamControl, a two-stage 2D-lifting text-to-3D framework that first optimizes a coarse NeRF as a 3D self-prior (with adaptive viewpoint sampling and a boundary integrity termination metric) and then refines textures with ControlNet-guided control-based score distillation using a conditional LoRA and weighted score.
Problem
Text-to-3D generation via 2D lifting (score distillation) suffers from inconsistent geometry (the Janus problem), caused by viewpoint bias in 2D diffusion models and overfitting of the SDS optimization.
Approach
Treat a coarse NeRF optimized before overfitting as a self-generated 3D prior, align camera sampling with the 2D diffusion model's view distribution, terminate optimization using a boundary density-gap metric, and use the prior's edge/normal renderings as ControlNet and conditional-LoRA conditions with a weighted (fused) score to refine texture while keeping geometry.
Key results
- On 30 prompts, DreamControl achieves JR 10.00%, PickScore 27.46%, and CLIP-Score 28.14, outperforming DreamFusion-IF, Magic3D, ProlificDreamer, Zero-1-to-3, and MVDream on all metrics.
- Magic3D and ProlificDreamer show JR of 53.33% and 56.67% respectively, indicating that higher PickScore tends to coincide with more Janus artifacts.
- Adaptive viewpoint sampling delays the appearance of extra legs in the corgi ablation until about 1,000 iterations, versus earlier under uniform sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DreamControl instead of the strongest reported baseline | Text-to-3D generation · Text-to-3D (30 prompts) | PS (%): 17.70 -> 27.46 | Helped |
| Use DreamControl instead of the strongest reported baseline | Text-to-3D generation · Text-to-3D (30 prompts) | CS (%): 26.69 -> 28.14 | Helped |
| Use DreamControl instead of the strongest reported baseline | Text-to-3D generation · Text-to-3D (30 prompts) | JR (%): 10.00 -> 10.00 | No clear effect |
| Add adaptive viewpoint sampling (ablation) | DreamControl · Corgi generation | Geometry consistency | Helped |
| Add boundary-integrity-based optimization termination (ablation) | DreamControl · Corgi generation | Geometry consistency | Helped |
| Replace camera-pose conditioning with normal-map conditioning in LoRA (ablation) | DreamControl · Corgi generation | Generation quality | Helped |
| Add a gradually increasing weighted LoRA score (ablation) | DreamControl · Corgi generation | Optimization stability | Helped |
Surprising findings
- Higher texture-aesthetic scores in baseline methods coincided with more Janus artifacts, suggesting overfitting exacerbates 3D inconsistency.
- Zero-1-to-3 and MVDream, which use 3D priors, produce consistent geometry but lower texture quality and text alignment.
Limitations
- May fail when prior renderings look similar across opposite views, causing ControlNet to give incorrect guidance and multi-face outputs.
- Relies on baselines reproduced in threestudio rather than official implementations, which may differ from originals.
- Multi-leg artifacts persist with adaptive sampling, so early termination is still needed.
Experimental setup
Datasets: DTU MVS dataset (multi-view images for user-guided generation example)
Benchmarks: 30 text prompts from DreamFusion, Magic3D, and ProlificDreamer galleries
Metrics: Janus problem occurrence rate (JR) PickScore (PS) CLIP-Score (CS)
Baselines: DreamFusion-IF Magic3D ProlificDreamer Zero-1-to-3 MVDream Latent-NeRF Fantasia3D
Builds on: DreamFusion ProlificDreamer Magic3D ControlNet threestudio
Closest papers by meaning
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.89)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.89)
- DreamComposer: Controllable 3D Object Generation via Multi-View Conditions (CVPR 2024 · similarity 0.89)
- DreamFlow: High-quality text-to-3D generation by Approximating Probability Flow (ICLR 2024 · similarity 0.88)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.88)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.