Research map / 3D vision / Face and shape
High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spatially-Varying Expression (SVE) conditioning for a NeuS-based deformable NeRF, generated by an MLP from 3DMM expression and positional features, plus coarse-to-fine training (depth-based geometry initialization and adaptive importance sampling) for monocular 3D head avatars.
Problem
Monocular NeRF-based 3D head avatars struggle to retain intricate facial expression details and accurate geometry because the radiance field is conditioned only on a global, spatially constant 3DMM expression vector.
Approach
Generate a position-dependent expression code by fusing each query point's spatial features with the global 3DMM expression via an MLP generation network, use it to condition both a tiny deformation MLP and a NeuS radiance field, and train with depth-supervised geometry initialization and loss-guided adaptive importance sampling over 19 semantic regions.
Key results
- On self-reenactment averaged over nine subjects, the method reaches PSNR 27.748 and LPIPS 0.0925, versus 25.360 PSNR and 0.145 LPIPS for the best baseline IMAvatar (LPIPS best baseline NeRFace 0.122).
- Ablation: removing SVE drops PSNR from 24.335 to 24.098; removing adaptive importance sampling drops PSNR to 24.011; removing depth supervision drops PSNR to 23.707.
- Compression of the SVE (64-dim to lower dim) improves over the uncompressed SVE (LPIPS 0.0911 vs 0.0923 in ablation table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Spatially-Varying Expression conditioning (ablation) | 3D head avatar NeRF · Ablation study | PSNR: 24.098 -> 24.335 | Helped |
| Compress the Spatially-Varying Expression code (ablation) | SVE generation network · Ablation study | PSNR: 24.090 -> 24.335 | Helped |
| Add coarse-stage depth supervision (ablation) | 3D head avatar training · Ablation study | PSNR: 23.707 -> 24.335 | Helped |
| Restrict depth supervision to the coarse stage instead of using it throughout training (ablation) | 3D head avatar training · Ablation study | PSNR: 24.069 -> 24.335 | Helped |
| Add adaptive importance sampling instead of random pixel sampling (ablation) | 3D head avatar training · Ablation study | PSNR: 24.011 -> 24.335 | Helped |
| Use Spatially-Varying Expression conditioning and coarse-to-fine training | 3D head avatar reconstruction · Self-reenactment, average of nine subjects | PSNR: 25.360 -> 27.748 | Helped |
Surprising findings
- Reducing the dimensionality of the expression code avoids overfitting, so compressed SVE outperforms the high-dimensional uncompressed version despite having less information.
Limitations
- Generalization is limited by training data distribution; teeth are hard to generate when mouth-opening frames are under 5% of video.
- Evaluated on a small set of monocular sequences (nine subjects in the self-reenactment average).
Experimental setup
Datasets: IMAvatar dataset NeRFace dataset self-collected iPhone 12 monocular sequences
Benchmarks: self-collected iPhone 12 dataset IMAvatar dataset NeRFace dataset
Metrics: L1 PSNR SSIM LPIPS MAE
Baselines: NeRFace IMAvatar PointAvatar
Builds on: NeRF NeuS Deformable NeRF IMAvatar NeRFace
Closest papers by meaning
- Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians (CVPR 2024 · similarity 0.90)
- GPAvatar: High-fidelity Head Avatars by Learning Efficient Gaussian Projections (CVPR 2025 · similarity 0.90)
- DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars (CVPR 2024 · similarity 0.90)
- Efficient 3D Implicit Head Avatar with Mesh-anchored Hash Table Blendshapes (CVPR 2024 · similarity 0.89)
- 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024 · similarity 0.89)
- GPAvatar: Generalizable and Precise Head Avatar from Image(s) (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.