Research map / 3D vision / Face and shape
DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 3D head avatar by using a ControlNet-conditioned Stable Diffusion renderer driven by rasterized NPHM meshes, TriPlane-rigged learnable features, and cross-attention on NPHM expression codes.
Problem
Creating photorealistic, animatable 3D head avatars with precise control over pose, expression and viewpoint while keeping temporal and view consistency, which 2D generative models lack and 3D avatar methods often render with less realism.
Approach
Deferred Diffusion: a latent diffusion model adapted via ControlNet to render NPHM proxy-geometry buffers, with person-specific learnable features tied to the NPHM canonical space through TriPlanes, plus direct expression-code conditioning via added cross-attention layers.
Key results
- Self-reenactment on NeRSemble (8 persons): PSNR 24.9, LPIPS 0.081, JOD 7.55, best among compared methods including DNR (LPIPS 0.226) and DiffusionRig (LPIPS 0.220).
- User study (735 responses, 49 participants): visual quality 4.02 vs 3.06 for the next best (DNR+GAN) on a 1-5 scale.
- Ablation: removing diffusion raises LPIPS from 0.074 to 0.133 and removing the pre-trained LDM prior gives slightly lower PSNR (24.5 vs 25.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diffusion instead of single-pass output prediction (ablation) | DiffusionAvatars · NeRSemble self-reenactment | LPIPS: 0.133 -> 0.074 | Helped |
| Use NPHM instead of FLAME (ablation) | DiffusionAvatars · NeRSemble self-reenactment | AKD: 2.46 -> 1.91 | Helped |
| Add direct expression conditioning (ablation) | DiffusionAvatars · NeRSemble self-reenactment | AKD: 1.98 -> 1.91 | Helped |
| Use the pre-trained LDM prior instead of training from scratch (ablation) | DiffusionAvatars · NeRSemble self-reenactment | LPIPS: 0.078 -> 0.074 | Hurt |
| Add TriPlane spatial features (ablation) | DiffusionAvatars · NeRSemble self-reenactment | JOD: 7.77 -> 7.85 | Helped |
| Replace TriPlanes with spherical feature mapping (ablation) | DiffusionAvatars · NeRSemble self-reenactment | JOD: 7.77 -> 7.77 | No clear effect |
Surprising findings
- Using FLAME instead of NPHM as proxy geometry gives consistently worse performance, especially on AKD.
Limitations
- Lighting is baked into generated images, so relighting is not supported.
- Denoising loop is too compute-intensive for real-time use.
- Requires per-person training on multi-view videos with fitted NPHM meshes.
Experimental setup
Datasets: NeRSemble
Benchmarks: NeRSemble
Metrics: PSNR LPIPS JOD AKD AED APD CSIM user study VQ user study DF
Baselines: NeRFace DiffusionRig Deferred Neural Rendering DNR+GAN Mixture of Volumetric Primitives
Builds on: Stable Diffusion ControlNet IPAdapter Deferred Neural Rendering NPHM
Closest papers by meaning
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.91)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.90)
- Efficient 3D Implicit Head Avatar with Mesh-anchored Hash Table Blendshapes (CVPR 2024 · similarity 0.90)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.88)
- IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular VideosC (CVPR 2025 · similarity 0.88)
- Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.