Research map / 3D vision / Face and shape
FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaceTalk, a transformer-based latent diffusion model that generates audio-driven 3D head expression sequences in the latent space of NPHM, trained on a dataset of temporally optimized NPHM expressions fit to NeRSemble recordings.
Problem
Audio-driven 3D talking-head animation has been limited to template meshes or 3DMMs, which lack fine-scale detail (hair, ears, eye blinks, wrinkles) and produce little upper-face motion; no paired audio–volumetric head expression data existed.
Approach
Couple wav2vec 2.0 audio features with a diffusion transformer decoder that denoises whole NPHM expression sequences, using an audio-frame alignment mask, FiLM timestep conditioning, classifier-free guidance, and magnitude-based expression augmentation; the training pairs are built by temporally regularized NPHM expression fitting to NeRSemble multi-view video.
Key results
- Achieves LSE-D 11.27 vs. 11.61 for the best baseline (Imitator) and FID 40.69 vs. 201.31 for EmoTalk on the hybrid test set (Table 1).
- User study with 40 participants shows FaceTalk preferred for overall quality (75.56%), lip sync (71.11%) and facial realism (73.89%) over baselines.
- Ablations show removing the expression-audio alignment mask raises LSE-D to 13.72 and removing diffusion yields zero diversity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add expression augmentation (ablation) | FaceTalk · Hybrid test set | LSE-D: 11.6229 -> 11.2737 | Helped |
| add expression augmentation (ablation) | FaceTalk · Hybrid test set | Diversity: 1.61e-8 -> 0.34 | Helped |
| add facial smoothing (ablation) | FaceTalk · Hybrid test set | LSE-D: 11.3488 -> 11.2737 | Helped |
| add facial smoothing (ablation) | FaceTalk · Hybrid test set | Diversity: 0.34 -> 0.34 | No clear effect |
| add FiLM conditioning (ablation) | FaceTalk · Hybrid test set | LSE-D: 12.029 -> 11.2737 | Helped |
| add FiLM conditioning (ablation) | FaceTalk · Hybrid test set | Diversity: 0.006 -> 0.34 | Helped |
| add expression-audio alignment (ablation) | FaceTalk · Hybrid test set | LSE-D: 13.720 -> 11.2737 | Helped |
| add expression-audio alignment (ablation) | FaceTalk · Hybrid test set | Diversity: 5.17e-8 -> 0.34 | Helped |
| add diffusion (ablation) | FaceTalk · Hybrid test set | LSE-D: 11.3217 -> 11.2737 | No clear effect |
| add diffusion (ablation) | FaceTalk · Hybrid test set | Diversity: 0.000 -> 0.34 | Helped |
Surprising findings
- Without diffusion the model produces zero diversity (0.000), indicating deterministic regression collapses expressions.
Limitations
- Diffusion requires multiple denoising steps, limiting real-time application.
- Only synthesizes expression codes; identity is not generated and must be given.
- Relies on the NeRSemble-fitted dataset with fixed known identities.
Experimental setup
Datasets: NeRSemble Vocaset LJSpeech
Benchmarks: Vocaset LJSpeech
Metrics: LSE-D FID KID FIQA VQA diversity score user preference
Baselines: VOCA MeshTalk FaceFormer CodeTalker Imitator EmoTalk EMOTE
Builds on: NPHM wav2vec 2.0 classifier-free guidance FiLM NeRSemble
Closest papers by meaning
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.91)
- MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation (AAAI 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.90)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.90)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.