Research map / Image generation / Text-to-image diffusion
Diverse and Stable 2D Diffusion Guided Text to 3D Generation with Noise Recalibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies diversity and degeneration problems of SDS loss in text-to-3D and proposes NR-SDS, which uses single-noise training plus a learnable noise recalibrated against a fixed anchor noise to improve diversity and stability of NeRF generation.
Problem
Score distillation sampling (SDS) for text-to-3D generation produces low diversity across random seeds and degenerates (loses texture detail) during long training.
Approach
Replace random per-step noise in SDS with a single learnable "moving" noise per timestep, and add a Noise Recalibration loss that aligns the moving noise's prediction at high guidance (w=100) with a fixed anchor noise's prediction at w=1, so the NeRF can be trained at high guidance without collapsing.
Key results
- User study: NR-SDS preferred over SDS baseline for quality in 80% and for diversity in 75% of comparisons (per Table 1 as extracted).
- NR-SDS preferred over DreamFusion at 67% and attains comparable preference against VSD (50%).
- NR-SDS needs about 60% of the running memory of VSD in 2D-space comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use single-noise training instead of sampling noise across the Gaussian space (ablation) | SDS for text-guided 3D generation · Text prompt “a cat” | generation diversity | Helped |
| Add the Noise Recalibration loss to single-noise training (ablation) | NR-SDS · Text-guided 3D generation | resistance to degeneration during training | Helped |
| Use NR-SDS instead of SDS | Text-guided 3D generation · User study: diversity comparison | preference score: 80% | Helped |
| Use NR-SDS instead of SDS | Text-guided 3D generation · User study: quality comparison | preference score: 75% | Helped |
| Use NR-SDS instead of Dreamfusion | Text-guided 3D generation · User study | preference score: 67% | Helped |
| Use NR-SDS instead of VSD | Text-guided 3D generation · User study | preference score: 50% | No clear effect |
Surprising findings
- Standard SDS guidance weight of 100 is needed to learn good shapes, yet this high weight causes degeneration; the paper argues the diffusion model is only calibrated at w=1.
Limitations
- Often falls short in learning object shapes and suffers from multi-head Janus problem
- Results shown are NeRF-only without DMTet finetuning
Experimental setup
Metrics: user preference score
Baselines: DreamFusion SDS VSD
Builds on: DreamFusion SDS loss Instant-NGP Stable Diffusion VSD loss
Closest papers by meaning
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.89)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.89)
- Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors (CVPR 2024 · similarity 0.89)
- DreamFlow: High-quality text-to-3D generation by Approximating Probability Flow (ICLR 2024 · similarity 0.88)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.88)
- Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.