Research map / Image generation / Editing and style transfer
Controllable 3D Face Generation with Conditional Style Code Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEx-Face, which inverts faces into EG3D style codes with pose-guided inversion and then trains a diffusion denoiser over those codes conditioned on CLIP text and EMOCA expression codes for controllable 3D face generation.
Problem
Generating many photorealistic 3D faces efficiently and controllably from text and expression conditions, since existing text-to-3D methods rely on slow per-object optimization and parametric-model approaches lack realism.
Approach
Encoder-based 3D GAN inversion into EG3D style codes, made view-invariant via a pose-guided mapping trained on synthetic multi-view renders, followed by a diffusion model that denoises style codes conditioned on decoupled text and expression cross-attention, with data augmentation (HairCLIP edits and captioning) to expand paired visual-language data.
Key results
- Conditional generation FID of 52.91 and CLIP score of 24.83, versus 111.36 / 24.51 for CD and 157.81 / 24.27 for TediGAN.
- GAN inversion FID of 30.15 versus 33.97 for e4e and 34.59 for Style Transformer, with lower ID and MSE than the baselines in the reported table.
- Inference takes about 0.10 s per sample for inversion versus about 120 s for PTI optimization; DDIM sampling with 100 steps takes about 5.1 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the refinement module to Re-PoI (ablation) | TEx-Face GAN inversion · CelebA-HQ test | MSE: 0.0404 -> 0.0252 | Helped |
| Add the refinement module to Re-PoI (ablation) | TEx-Face GAN inversion · CelebA-HQ test | LPIPS: 0.2128 -> 0.1721 | Helped |
| Add the refinement module to Re-PoI (ablation) | TEx-Face GAN inversion · CelebA-HQ test | ID: 0.2156 -> 0.1788 | Helped |
| Add the refinement module to Re-PoI (ablation) | TEx-Face GAN inversion · CelebA-HQ test | FID: 35.90 -> 30.15 | Helped |
| Use separate cross-attention layers for text and expression instead of one shared layer (ablation) | TEx-Face conditional generation · Conditional 3D face generation | Expression score: 58.38 -> 59.70 | Helped |
| Use separate cross-attention layers for text and expression instead of one shared layer (ablation) | TEx-Face conditional generation · Conditional 3D face generation | CLIP score: 20.62 -> 21.86 | Helped |
| Replace Style Transformer with TEx-Face inversion | EG3D GAN inversion · CelebA-HQ test | MSE: 0.0259 -> 0.0252 | Helped |
| Replace Style Transformer with TEx-Face inversion | EG3D GAN inversion · CelebA-HQ test | LPIPS: 0.1677 -> 0.1721 | Hurt |
| Replace Style Transformer with TEx-Face inversion | EG3D GAN inversion · MEAD novel-view synthesis | MSE: 0.168 -> 0.048 | Helped |
| Replace Style Transformer with TEx-Face inversion | EG3D GAN inversion · MEAD novel-view synthesis | LPIPS: 0.477 -> 0.189 | Helped |
| Replace Style Transformer with TEx-Face inversion | EG3D GAN inversion · MEAD novel-view synthesis | ID: 0.375 -> 0.244 | Helped |
| Use TEx-Face instead of CD | Conditional face generation · Conditional face generation | FID: 111.36 -> 52.91 | Helped |
Surprising findings
- Style Transformer has the lowest LPIPS in GAN inversion but worse results on the other metrics than the proposed method.
- Expression confidence is low for Sad, Disgust, and Anger (35.92 to 53.97), which the authors attribute partly to data imbalance.
Limitations
- Generation quality depends heavily on EG3D and the inversion model.
- Expression training data is imbalanced, so the model tends to favor majority expressions such as happy.
- Data augmentation quality depends on the quality of the manipulation and captioning models.
Experimental setup
Datasets: FFHQ CelebA-HQ CelebA-Dialog MEAD
Benchmarks: CelebA-HQ test MEAD CelebA-Dialog
Metrics: MSE LPIPS ID similarity FID CLIP score expression score user study accuracy
Baselines: e4e Style Transformer PTI TediGAN CD
Builds on: EG3D StyleGAN2 EMOCA CLIP Denoising diffusion probabilistic models
Closest papers by meaning
- Diffusion-driven GAN Inversion for Multi-Modal Face Image Generation (CVPR 2024 · similarity 0.88)
- Text-Guided 3D Face Synthesis - From Generation to Editing (CVPR 2024 · similarity 0.87)
- Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization (ICML 2024 · similarity 0.87)
- Text-conditional Attribute Alignment across Latent Spaces for 3D Controllable Face Image Synthesis (CVPR 2024 · similarity 0.86)
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.86)
- Data Synthesis with Diverse Styles for Face Recognition via 3DMM-Guided Diffusion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.