Research map / 3D vision / Face and shape
Learning Dynamic Tetrahedra for High-Quality Talking Head Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynTet, a hybrid explicit-implicit representation that extends tetrahedral SDF grids with Marching Tetrahedra to dynamic, topology-consistent head meshes for real-time talking head synthesis from video.
Problem
Personalized talking head synthesis from short videos suffers from visual artifacts, jitter, and motionless mouths in implicit NeRF-based methods, which lack explicit geometric constraints for complex facial deformations.
Approach
Extends static tetrahedral SDF representations to dynamic heads by decoupling topology (SDF from a network) from deformation (offsets scaled by learned per-vertex elastic scores conditioned on 3DMM expression), plus a 3DMM-based canonical space and geometry losses, decoding meshes with Marching Tetrahedra and rendering via a differentiable rasterizer.
Key results
- Self-driven reconstruction: DynTet reaches PSNR 35.15 and AUE 1.833 vs ER-NeRF 32.50 and 2.773, running at 46 FPS at 512x512.
- DynTet† (with LPIPS supervision) achieves LPIPS 0.0223 and FID 4.72, better than ER-NeRF's 0.0345 and 6.44.
- Cross-driven lip sync on Testset A: DynTet achieves the lowest AUE (3.672) among compared methods, but a lower Sync score (5.055) than Wav2Lip (8.394).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3DMM supervision (ablation) | DynTet · Self-driven head reconstruction | LPIPS: 0.0686 -> 0.0223 | Helped |
| Add canonical projection (ablation) | DynTet · Self-driven head reconstruction | LPIPS: 0.0529 -> 0.0223 | Helped |
| Add elastic score (ablation) | DynTet · Self-driven head reconstruction | LMDm: 2.336 -> 2.284 | Helped |
| Use frequency encoding instead of hash encoding for geometry (ablation) | DynTet · Talking-head mesh quality | mesh quality | Helped |
| Set appearance mapping middle neurons to 32 instead of the default 256 (ablation) | DynTet · Self-driven head reconstruction | LPIPS: 0.0219 -> 0.0539 | Hurt |
| Set appearance mapping middle neurons to 64 instead of the default 256 (ablation) | DynTet · Self-driven head reconstruction | LPIPS: 0.0219 -> 0.0254 | Hurt |
| Set appearance mapping middle neurons to 128 instead of the default 256 (ablation) | DynTet · Self-driven head reconstruction | LPIPS: 0.0219 -> 0.0237 | Hurt |
| Set appearance mapping middle neurons to 512 instead of the default 256 (ablation) | DynTet · Self-driven head reconstruction | FPS: 46 -> 23 | Hurt |
| Add overall LPIPS supervision during training (ablation) | DynTet · Self-driven head reconstruction | LPIPS: 0.0619 -> 0.0223 | Helped |
| Use Audio2Exp-generated coefficients to drive DynTet instead of frame-driven coefficients | DynTet · Cross-driven lip synchronization, Testset A | Sync: 5.055 -> 7.055 | Helped |
| Use Audio2Exp-generated coefficients to drive DynTet instead of frame-driven coefficients | DynTet · Cross-driven lip synchronization, Testset B | Sync: 6.401 -> 7.335 | Helped |
Surprising findings
- Pre-trained 2D lip-sync methods (Wav2Lip, PC-AVS) achieve higher SyncNet confidence than DynTet, which is the best in AUE but not in Sync.
- Replacing frequency encoding with hash encoding produces cluttered meshes, so low-frequency encoding is preferred for geometry.
Limitations
- Lip-sync Sync score lags 2D pre-trained methods such as Wav2Lip and PC-AVS.
- Back of meshes may have defects due to limited frontal training data.
- Requires a several-minute person-specific video for training.
Experimental setup
Datasets: publicly-released talking head video dataset (four videos, used by AD-NeRF/ER-NeRF) SynObama demo clips NVP demo clips
Benchmarks: self-driven head reconstruction test sets (four videos) cross-driven lip synchronization Testset A and B
Metrics: PSNR LPIPS FID CSIM LMDm LMDe AUE Sync training time FPS
Baselines: Wav2Lip PC-AVS AD-NeRF RAD-NeRF ER-NeRF NVP SynObama SadTalker
Builds on: nvdiffrec Marching Tetrahedra 3DMM ER-NeRF DMTet
Closest papers by meaning
- Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture (CVPR 2025 · similarity 0.87)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.86)
- LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example (CVPR 2024 · similarity 0.86)
- Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians (CVPR 2024 · similarity 0.85)
- 3D Gaussian Head Avatars with Expressive Dynamic Appearances by Compact Tensorial Representations (CVPR 2025 · similarity 0.85)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.