Research map / Image generation / Editing and style transfer
Text-Guided 3D Face Synthesis - From Generation to Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaceG2E, a text-guided pipeline that generates 3D faces via decoupled geometry and texture SDS (with ControlNet depth conditioning and a fine-tuned RGB/YUV texture diffusion prior) and supports sequential editing via InstructPix2Pix SDS with a self-guided UV consistency regularization.
Problem
Existing text-guided 3D face methods focus on direct generation, yielding loss of geometric detail, geometry-texture misalignment, and cannot support precise, sequential editing without unintended changes to other attributes.
Approach
Decouple generation into a texture-less geometry phase and a depth-ControlNet-conditioned texture phase with a fine-tuned RGB+YUV texture prior, and perform editing with InstructPix2Pix SDS regularized by a UV-domain consistency weight derived from cross-attention scores of the edit token.
Key results
- CLIP score of 36.95 for generation and 35.50 for editing, versus 34.85/33.73 for TADA and 33.22/33.14 for DreamFace.
- Ranking-1 of 80% (generation) and 70% (editing) for FaceG2E in CLIP evaluation.
- User study with 100 participants on 10 generation and 10 editing examples found FaceG2E preferred over all compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FaceG2E instead of TADA | Text-guided 3D face generation · Generation | CLIP score: 34.85 -> 36.95 | Helped |
| Use FaceG2E instead of TADA | Text-guided 3D face editing · Editing | CLIP score: 33.73 -> 35.50 | Helped |
| Add geometry-centric SDS using texture-less rendering (ablation) | FaceG2E geometry generation · 3D face geometry | geometric detail | Helped |
| Replace depth-ControlNet with standard Stable Diffusion for geometry-aware texture SDS (ablation) | FaceG2E texture generation · 3D face texture | geometry-texture alignment | Hurt |
| Add texture prior SDS to geometry-aware texture content SDS (ablation) | FaceG2E texture generation · 3D face texture | texture quality | Helped |
| Add YUV texture prior SDS (ablation) | FaceG2E texture generation · 3D face texture | brightness uniformity | Helped |
| Add consistency-preservation regularization (ablation) | FaceG2E face editing · 3D face editing | consistency of unaffected attributes | Helped |
| Add self-guided consistency weighting (ablation) | FaceG2E face editing · 3D face editing | editing effectiveness | Helped |
Limitations
- Geometry representation limited to facial skin; cannot generate hair or accessories.
- Sequential editing substantially increases time per edit round.
- Evaluation relies on a small set of 20 prompts and CLIP score plus a user study.
Experimental setup
Datasets: facial texture dataset (500 textures, scanned and synthesized)
Benchmarks: 20 text prompts (generation and editing) for CLIP score
Metrics: CLIP score CLIP Ranking-1 user preference
Baselines: Describe3D DreamFace TADA
Builds on: DreamFusion DreamFace InstructPix2Pix ControlNet TADA
Closest papers by meaning
- Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization (ICML 2024 · similarity 0.89)
- DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors (CVPR 2024 · similarity 0.88)
- Controllable 3D Face Generation with Conditional Style Code Diffusion (AAAI 2024 · similarity 0.87)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.87)
- PaintHuman: Towards High-Fidelity Text-to-3D Human Texturing via Denoised Score Distillation (AAAI 2024 · similarity 0.87)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.