Research map / Image generation / Editing and style transfer
Diffusion-driven GAN Inversion for Multi-Modal Face Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Links a pre-trained ControlNet diffusion model's multi-modal features to the latent space of pre-trained StyleGAN/EG3D via a mapping network and attention-based style modulation, enabling text+mask/scribble face generation in 2D and 3D.
Problem
Multi-modal (text plus semantic mask or scribble) face image generation where GAN-inversion methods align poorly with inputs and diffusion models produce unrealistic faces.
Approach
Uses the middle and decoder features and cross-attention maps of a ControlNet denoising U-Net as an encoder into a pre-trained GAN's latent space: a map2style-based mapping network produces latent codes, and an attention-weighted style modulation network scales and shifts them, trained across multiple denoising steps.
Key results
- On CelebAMask-HQ with text+mask, the 2D variant reaches FID 46.68, LPIPS 0.30, ID 0.76, ACC 83.41, mIoU 43.82, versus TediGAN FID 54.83 and UaC FID 45.87.
- The 3D variant (EG3D) reaches FID 44.91 and ID 0.78, improving over IDE-3D (FID 39.05 but ID 0.54, ACC 47.07).
- Ablation on 3D generation shows the attention-based style modulation lifts ACC from 81.09 (M only, with attention removed) to 83.05 in the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add style modulation network T to mapping network M (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | FID: 62.08 -> 44.91 | Helped |
| Add style modulation network T to mapping network M (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | ACC: 81.09 -> 83.05 | Helped |
| Add cross-attention maps At and ¯At to style modulation (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | FID: 54.27 -> 44.91 | Helped |
| Add cross-attention maps At and ¯At to style modulation (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | ID: 0.58 -> 0.78 | Helped |
| Add cross-attention maps At and ¯At to style modulation (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | ACC: 80.58 -> 83.05 | Helped |
| Train with both Id0 and Igt rather than Igt only (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | FID: 61.60 -> 44.91 | Helped |
| Train with both Id0 and Igt rather than Id0 only (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | ACC: 80.58 -> 83.05 | Helped |
| Use both Id0 and Igt for training rather than Id0 only (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | LPIPS: 0.31 -> 0.28 | Helped |
| Use both Id0 and Igt for training rather than Igt only (ablation) | Diffusion-driven GAN inversion · CelebAMask-HQ, text + semantic mask | ID: 0.62 -> 0.78 | Helped |
| Use the proposed method instead of ControlNet | Face generation from text and scribble map · CelebAMask-HQ, text + scribble map | FID: 93.26 -> 55.60 | Helped |
| Use the proposed method instead of ControlNet | Face generation from text and scribble map · CelebAMask-HQ, text + scribble map | LPIPS: 0.52 -> 0.32 | Helped |
| Use the proposed method instead of UaC | Face generation from text and semantic mask · CelebAMask-HQ, text + semantic mask | ID: 0.32 -> 0.76 | Helped |
Surprising findings
- IDE-3D achieves the best FID (39.05) despite weaker identity, ACC, and mIoU scores than the proposed method.
Limitations
- Cannot transfer extremely distinct artistic style attributes to the photo-realistic GAN domain in 3D style transfer.
- 3D-aware style transfer relies on Id0 from the U-Net instead of ground-truth images.
- Evaluated only on CelebAMask-HQ faces with a limited set of conditions (semantic mask, scribble).
Experimental setup
Datasets: CelebAMask-HQ
Benchmarks: CelebAMask-HQ
Metrics: FID LPIPS SSIM ID similarity ACC mIoU
Baselines: TediGAN IDE-3D Unite and Conquer ControlNet Collaborative diffusion
Builds on: ControlNet pSp StyleGAN EG3D
Closest papers by meaning
- Controllable 3D Face Generation with Conditional Style Code Diffusion (AAAI 2024 · similarity 0.88)
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation (ICML 2025 · similarity 0.88)
- DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors (CVPR 2024 · similarity 0.87)
- OSDFace: One-Step Diffusion Model for Face Restoration (CVPR 2025 · similarity 0.87)
- In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing (CVPR 2024 · similarity 0.86)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.