Research map / 3D vision / Face and shape
MonoNPHM: Dynamic Head Reconstruction from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MonoNPHM, a neural-field parametric head model with a geometry-conditioned latent appearance space and hyper-dimensional backward deformations, and uses it for photometric tracking of 3D heads from monocular RGB video.
Problem
Dynamic 3D head reconstruction from monocular RGB videos is heavily under-constrained, and existing parametric priors (3DMMs, NeRF-based avatars) either have limited expressiveness or do not handle monocular video tracking with full heads.
Approach
Condition the texture field on hidden geometry features so RGB gradients flow into geometry latent codes, augment the backward deformation field with hyper-dimensions for topologically challenging expressions, and add a landmark loss from numerically inverted backward deformations at facial anchor points, all within an SDF-based volumetric rendering tracker.
Key results
- MonoNPHM achieves L1-Chamfer 0.0024 m, normal consistency 0.940 and Recall@2.5mm 0.785, versus 0.0030 for the best baseline MICA tracker (Chamfer).
- The paper reports about a 20% Chamfer improvement over its strongest baselines.
- Ablations show removing spherical harmonics, the landmark loss, or the geometry-to-color feature channel each degrades the metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sphere tracing with NeuS-style volume rendering (ablation) | MonoNPHM tracking · 20 Kinect sequences | L1-Ch.: 0.0033 -> 0.0024 | Helped |
| add spherical harmonics shading (ablation) | MonoNPHM tracking · 20 Kinect sequences | L1-Ch.: 0.0028 -> 0.0024 | Helped |
| add landmark loss (ablation) | MonoNPHM tracking · 20 Kinect sequences | L1-Ch.: 0.0027 -> 0.0024 | Helped |
| use the MonoNPHM architecture instead of NPHMapp (ablation) | MonoNPHM · 20 Kinect sequences | L1-Ch.: 0.0028 -> 0.0024 | Helped |
| use 65 anchors instead of 39 (ablation) | MonoNPHM · 20 Kinect sequences | Recall: 0.761 -> 0.785 | Helped |
| condition color on geometry features instead of canonical coordinates (ablation) | MonoNPHM · 20 Kinect sequences | L1-Ch.: 0.0028 -> 0.0024 | Helped |
| use local MLPs instead of a global MLP (ablation) | MonoNPHM · 20 Kinect sequences | L1-Ch.: 0.0026 -> 0.0024 | Helped |
| add hyper-dimensions to the backward deformation model (ablation) | MonoNPHM · topologically challenging expressions | geometry and color representation | Helped |
Surprising findings
- Sphere-tracing rendering and removing spherical harmonics or the landmark loss each reduce accuracy, but the landmark loss matters mainly for fast, large mouth movements rather than average metrics.
Limitations
- Tracking is slow (about 1.2 frames per minute) due to an unoptimized per-frame full optimization.
- Lighting and shadows are only approximated with 3-band spherical harmonics.
- Evaluation relies on a self-recorded set of 20 Kinect sequences from 5 participants.
Experimental setup
Datasets: NPHM dataset (473 identities) 20 self-recorded Kinect RGB-D sequences FFHQ
Benchmarks: 20 Kinect monocular RGB sequences (self-recorded)
Metrics: L1-Chamfer distance normal cosine similarity Recall@2.5mm
Baselines: DECA MICA tracker Neural Head Avatar (NHA) IMavatar HeadNeRF PanoHead
Builds on: NPHM NeuS IDR PhoMoH NeRF
Closest papers by meaning
- Learning Personalized High Quality Volumetric Head Avatars From Monocular RGB Videos (CVPR 2023 · similarity 0.88)
- Neural Head Avatars From Monocular RGB Videos (CVPR 2022 · similarity 0.87)
- MorpheuS: Neural Dynamic 360deg Surface Reconstruction from Monocular RGB-D Video (CVPR 2024 · similarity 0.87)
- MonoInstance: Enhancing Monocular Priors via Multi-view Instance Alignment for Neural Rendering and Reconstruction (CVPR 2025 · similarity 0.86)
- DPHMs: Diffusion Parametric Head Models for Depth-based Tracking (CVPR 2024 · similarity 0.85)
- Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.