Research map / 3D vision / Face and shape
UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
UltrAvatar generates animatable FLAME-based 3D avatars with PBR textures from a text prompt or single face image, using a self-attention-based diffuse color extraction model and an authenticity-guided texture diffusion model with photometric and edge guidance.
Problem
Text- or single-image-based 3D avatar generation suffers from over-smoothed SDS results lacking facial detail, and from unwanted lighting baked into textures, which prevents relighting and realistic PBR rendering.
Approach
Remove lighting from an input face by swapping query/key self-attention features with those from a semantic parsing mask (diffuse color extraction), then generate complete PBR UV textures with a fine-tuned texture diffusion model whose sampling is steered by photometric (rendering) and canny-edge gradient guidances, instead of SDS.
Key results
- UltrAvatar achieves FID 45.50 and KID 0.029 versus DreamFace's 76.70 and 0.061 and PanoHead's 48.64 and 0.039 in the comparison table.
- UltrAvatar reports CLIP score 0.301 ± 0.023, higher than DreamFace's 0.291 ± 0.020.
- Text-to-avatar generation takes about 2 minutes on a single Nvidia A6000, compared to 5 minutes for DreamFace.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UltrAvatar instead of PanoHead for avatar generation | Text-to-avatar generation · Text-to-avatar generation | FID: 48.64 vs 45.50 | Helped |
| Use UltrAvatar instead of ClipFace for avatar generation | Text-to-avatar generation · Text-to-avatar generation | KID: 0.032 vs 0.029 | Helped |
| Use UltrAvatar instead of DreamFace for avatar generation | Text-to-avatar generation · Text-to-avatar generation | CLIP Score: 0.291 ± 0.020 vs 0.301 ± 0.023 | Helped |
| Add photometric guidance to AGT-DM sampling (ablation) | UltrAvatar AGT-DM · Generated avatar textures | identity and texture alignment | Helped |
| Add edge guidance to AGT-DM sampling (ablation) | UltrAvatar AGT-DM · Generated avatar textures | facial detail | Helped |
| Add photometric and edge guidance together to AGT-DM sampling (ablation) | UltrAvatar AGT-DM · Generated avatar textures | identity and facial detail preservation | Helped |
| Replace image self-attention query and key features with semantic-mask features during denoising | UltrAvatar DCE · Input face image diffuse-color extraction | lighting removal | Helped |
Limitations
- PanoHead performance depends heavily on pre-processing accuracy, which sometimes fails
- Evaluation is limited to 40 text prompts and GPT-4V-based scoring rather than human study
- Training dataset is a proprietary annotated 3DScan store, and the NeRF-based PanoHead is limited in relighting
Experimental setup
Datasets: 3DScan dataset (online 3D face scans with PBR maps) FFHQ (evaluation reference)
Benchmarks: 40 text prompts evaluation with rendered views (FID/KID/CLIP score vs masked FFHQ)
Metrics: FID KID CLIP score
Baselines: DreamFace ClipFace Latent3d ClipMatrix Text2Mesh FlameTex PanoHead
Builds on: Stable Diffusion SDXL FLAME DDIM inversion Plug-and-Play diffusion features MICA EMOCA
Closest papers by meaning
- AvatarVerse: High-Quality & Stable 3D Avatar Creation from Text and Pose (AAAI 2024 · similarity 0.88)
- Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusion (CVPR 2025 · similarity 0.88)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.88)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.87)
- HERA: Hybrid Explicit Representation for Ultra-Realistic Head Avatars (CVPR 2025 · similarity 0.87)
- SimAvatar: Simulation-Ready Avatars with Layered Hair and Clothing (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.