Research map / 3D vision / Face and shape
DPHMs: Diffusion Parametric Head Models for Depth-based Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPHMs, a latent diffusion prior over NPHM identity and expression codes that regularizes volumetric head reconstruction and tracking from monocular depth sequences, plus a new Kinect dataset.
Problem
Volumetric head reconstruction and expression tracking from noisy, partial single-view depth sequences is under-constrained, and overparameterized neural parametric head models (NPHMs) overfit to such data.
Approach
Train DDPM-based diffusion models over the identity and expression latent codes of a backward-deformation NPHM, then use the diffusion noise predictions as score-based gradients to regularize latent optimization during depth-based tracking.
Key results
- On DPHM-Kinect, DPHMs achieve l2 error 1.465 mm vs 1.579 for NPHM and 5.251 for FLAME, with Recall@1.5mm 70.79 vs 65.95.
- On NerSemble, DPHMs reach l2 error 1.103 mm vs 1.201 for NPHM, with Recall@1.5mm 89.96 vs 84.73.
- Ablations show VAE prior and removing expression or identity diffusion regularizers each degrade accuracy relative to the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diffusion-prior regularization to NPHM | NPHM · DPHM-Kinect | ℓ2: 1.579 -> 1.465 | Helped |
| add diffusion-prior regularization to NPHM | NPHM · DPHM-Kinect | RC: 65.95 -> 70.79 | Helped |
| add diffusion-prior regularization to NPHM | NPHM · NerSemble | ℓ2: 1.201 -> 1.103 | Helped |
| add diffusion-prior regularization to NPHM | NPHM · NerSemble | RC: 84.73 -> 89.96 | Helped |
| use backward rather than forward deformations (ablation) | DPHM · DPHM-Kinect | ℓ2: 1.367 -> 1.146 | Helped |
| replace VAE priors with diffusion priors (ablation) | DPHM · DPHM-Kinect | ℓ2: 2.929 -> 1.146 | Helped |
| add expression diffusion regularization (ablation) | DPHM · DPHM-Kinect | ℓ2: 1.283 -> 1.146 | Helped |
| add expression diffusion regularization (ablation) | DPHM · DPHM-Kinect | NC: 90.60 -> 91.32 | No clear effect |
| add identity diffusion regularization (ablation) | DPHM · DPHM-Kinect | ℓ2: 1.159 -> 1.146 | Helped |
| add identity diffusion regularization (ablation) | DPHM · DPHM-Kinect | NC: 91.13 -> 91.32 | Helped |
Surprising findings
- A VAE prior over the same latents fails to ensure plausible identity reconstruction and expression tracking, performing worse than the simpler NPHM regularizer on some metrics.
Limitations
- Test-time latent optimization makes inference slow.
- Not real-time tracking.
- Evaluated on a relatively small set of 130 Kinect sequences and 20 NerSemble sequences.
Experimental setup
Datasets: NPHM head scans (4,760 scans, 239 identities) NerSemble multi-view video (single-view depth sequences via COLMAP)
Benchmarks: DPHM-Kinect NerSemble
Metrics: l2 distance (mm) normal consistency Recall@1.5mm Recall@3mm
Baselines: FLAME ImFace ImAvatar NPHM VAE prior
Builds on: NPHM DDPM IGR V-Poser
Closest papers by meaning
- DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars (CVPR 2024 · similarity 0.86)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.85)
- MonoNPHM: Dynamic Head Reconstruction from Monocular Videos (CVPR 2024 · similarity 0.85)
- Diffusion Prior Interpolation for Flexibility Real-World Face Super-Resolution (AAAI 2025 · similarity 0.83)
- Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking (CVPR 2024 · similarity 0.83)
- Synthetic Prior for Few-Shot Drivable Head Avatar Inversion (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.