Research map / 3D vision / 3D generation
Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E^3-FaceNet, a single-stage text-to-3D-aware face generator built on StyleNeRF that injects CLIP text features into the style code, with a Style Code Enhancer and a geometric (3D location + normal) regularization for multi-view consistency and fast text-guided 3D face generation and editing.
Problem
Text-to-3D-aware face generation and manipulation is slow (multi-stage pipelines, test-time optimization) and suffers from poor multi-view consistency and weak text-attribute alignment when trained only on single-view 2D face data.
Approach
Direct end-to-end mapping from text to the style space of a pretrained StyleNeRF, enhanced by per-upsample-block cross-modal attention that predicts style code offsets (Style Code Enhancer, enabling interpolation-based editing) and a geometric regularization enforcing smoothness of 3D point locations and normals across neighboring pixels.
Key results
- On CelebAText, MVIC 0.8521 vs 0.7835 for Latent3D; five-view inference 0.64s vs 301.92s for Latent3D (about 471x faster).
- On MMCelebA, FID 12.46 vs 53.38 for StyleCLIP and 16.63 for PixelFace (best FID and KID).
- Text-driven manipulation user study: IP 69.92 vs 21.92 for Latent3D; EQ 75.92 vs 12.48 for ClipFace.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use E³-FaceNet instead of Describe3D | T3D face generation · CelebAText | MVIC: 0.7973 -> 0.8521 | Helped |
| use E³-FaceNet instead of Latent3D | T3D face generation · CelebAText | SA: 0.2688 -> 0.2623 | Hurt |
| use E³-FaceNet instead of Describe3D | T3D face generation · FFHQ-Text | MVIC: 0.8027 -> 0.8467 | Helped |
| use E³-FaceNet instead of Latent3D | T3D face generation · FFHQ-Text | SA: 0.2758 -> 0.2778 | Helped |
| use E³-FaceNet instead of Latent3D | T3D face generation · five-view generation | inference time: 301.92s -> 0.64s | Helped |
| use E³-FaceNet instead of Latent3D | face manipulation · user study | IP: 21.92 -> 69.92 | Helped |
| use E³-FaceNet instead of ClipFace | face manipulation · user study | EQ: 12.48 -> 75.92 | Helped |
| use E³-FaceNet instead of PixelFace | single-view face generation · MMCelebA | FID: 16.63 -> 12.46 | Helped |
| use E³-FaceNet instead of TediGAN | single-view face generation · MMCelebA | CLIP-Score: 0.2865 -> 0.2770 | Hurt |
| add geometric regularization and SCE (ablation) | E³-FaceNet · MMCelebA | FID: 13.44 -> 12.46 | Helped |
| add location regularization (ablation) | E³-FaceNet · CelebAText | MVIC: 0.7960 -> 0.8431 | Helped |
| add normal regularization (ablation) | E³-FaceNet · CelebAText | MVIC: 0.7960 -> 0.8307 | Helped |
Surprising findings
- On the MMCelebA CLIP-Score, TediGAN (0.2865) beats E^3-FaceNet (0.2770), and on CelebAText SA, Latent3D (0.2688) beats E^3-FaceNet (0.2623).
Limitations
- Semantic alignment (SA) is slightly worse than Latent3D on CelebAText and CLIP-Score is below TediGAN on MMCelebA.
- 3D baselines could not be trained on MMCelebA, so 3D comparisons are zero-shot on CelebAText and FFHQ-Text.
- Manipulation is evaluated only by a user study with 25 participants on 50 instructions.
Experimental setup
Datasets: MMCelebA CelebAText-HQ FFHQ-Text FFHQ
Benchmarks: MMCelebA CelebAText-HQ FFHQ-Text
Metrics: FID KID CLIP-Score MVIC SA inference time user study IP/SA/EQ
Baselines: Latent3D Describe3D ClipFace StyleCLIP TediGAN OpenFaceGAN PixelFace
Builds on: StyleNeRF NeRF EG3D TediGAN
Closest papers by meaning
- Text-Guided 3D Face Synthesis - From Generation to Editing (CVPR 2024 · similarity 0.89)
- Text-conditional Attribute Alignment across Latent Spaces for 3D Controllable Face Image Synthesis (CVPR 2024 · similarity 0.89)
- TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2026 · similarity 0.87)
- Controllable 3D Face Generation with Conditional Style Code Diffusion (AAAI 2024 · similarity 0.87)
- DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors (CVPR 2024 · similarity 0.86)
- 3D-Aware Face Editing via Warping-Guided Latent Direction Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.