Research map / Image generation / Text-to-image diffusion
DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Combines pretrained 3D GANs (EG3D/PanoHead/AG3D) with SDS guidance from text-to-image diffusion models for text-guided 3D domain adaptation and text-to-avatar generation, plus a progressive diffusion-based texture refinement stage.
Problem
Text-guided 3D domain adaptation and text-to-avatar generation suffer from inflexibility, instability, low fidelity, limited diversity, and poor texture quality when using existing 3D GAN adaptation or pure SDS-based text-to-3D methods.
Approach
Fine-tune the triplane generator of a pretrained EG3D-style 3D GAN with SDS loss from a diffusion model, regularized by a relative distance loss to preserve diversity, a diffusion-guided reconstruction loss for local editing, and a case-specific learnable triplane for avatars; then refine textures on an explicit mesh with ControlNet-guided progressive inpainting.
Key results
- Domain adaptation FID 28.5 vs 53.6 for StyleGAN-Fusion and 136.2 for StyleGAN-NADA*.
- User preference for text-to-avatar head generation 80.5% for the proposed method vs 11.7% for ProlificDreamer.
- Ablations show relative distance loss prevents generator collapse to a fixed output pattern while keeping stylization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffusionGAN3D instead of StyleGAN-Fusion | 3D domain adaptation on EG3D-face and EG3D-cat · 3D domain adaptation (EG3D-face and EG3D-cat) | FID: 53.6 -> 28.5 | Helped |
| Use DiffusionGAN3D instead of StyleGAN-Fusion | 3D domain adaptation on EG3D-face and EG3D-cat · 3D domain adaptation (EG3D-face and EG3D-cat) | image quality: 3.255 -> 3.758 | Helped |
| Use DiffusionGAN3D instead of DreamHuman | text-to-avatar body generation · Text-to-avatar body generation | user preference: 18.9% -> 66.6% | Helped |
| Add relative distance loss (ablation) | DiffusionGAN3D domain adaptation · 3D domain adaptation | generation diversity | Helped |
| Add diffusion-guided reconstruction loss instead of using ordinary reconstruction loss (ablation) | DiffusionGAN3D local editing · Local editing | target-region editing | Helped |
| Use progressive texture refinement instead of direct UV texture image-to-image (ablation) | DiffusionGAN3D texture refinement · Texture refinement | texture alignment | Helped |
| Use inpainting for progressive views instead of image-to-image translation (ablation) | DiffusionGAN3D texture refinement · Texture refinement | multi-view texture consistency | Helped |
| Use progressive texture refinement instead of MSE-based refinement (ablation) | DiffusionGAN3D texture refinement · Texture refinement | texture detail | Helped |
| Add adaptive blend module (ablation) | DiffusionGAN3D texture refinement · Texture refinement | texture smoothness | Helped |
| Add case-specific learnable triplane (ablation) | DiffusionGAN3D text-to-avatar generation · Text-to-avatar generation | generation accuracy | Helped |
| Add multi-scale total variation loss (ablation) | DiffusionGAN3D text-to-avatar generation · Text-to-avatar generation | result smoothness | Helped |
Limitations
- Framework only suitable for categories covered by the pretrained 3D GAN (e.g., head, human body)
- SDS gradients are unstable and can harm well-trained modules like super-resolution
Experimental setup
Datasets: FFHQ AFHQ
Benchmarks: FID on domain adaptation (EG3D-face, EG3D-cat)
Metrics: FID user study rating (text-correspondence, quality, diversity) user preference rate
Baselines: StyleGAN-NADA* StyleGAN-Fusion DreamFusion ProlificDreamer Magic3D DreamAvatar DreamHuman AvatarVerse
Builds on: EG3D DreamFusion StyleGAN-NADA StyleGAN-Fusion ControlNet
Closest papers by meaning
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.90)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.89)
- Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data (CVPR 2025 · similarity 0.89)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.89)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.89)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.