Research map / 3D vision / Face and shape
Hierarchically Controlled Deformable 3D Gaussians for Talking Head Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiCoDe, an audio-driven talking head framework built on deformable 3D Gaussians that controls deformation hierarchically via 3DMM (coarse) and facial landmarks (fine) using AdaIN, achieving higher quality and faster training/inference than NeRF-based methods.
Problem
Audio-driven talking head synthesis is costly for diffusion and NeRF methods; naively driving dense 3D Gaussians with sparse audio features yields poor expression control and artifacts.
Approach
Decouples identity from expression and controls sparse expression via a two-level hierarchy: Faceformer-predicted 3DMM expression parameters as coarse control and landmark-based attention plus AdaIN as fine control modulating Gaussian means and covariances, with a MediaPipe-based 3DGS initialization from monocular depth.
Key results
- On HDTF, HiCoDe reaches SSIM 0.983, PSNR 37.767, LPIPS 0.013 and LMD 1.819, versus GaussianTalker's 0.964 SSIM and 2.074 LMD.
- Training takes about 30 minutes versus 2h for GaussianTalker and 8h for RAD-NeRF; inference runs at 157 FPS versus 98 for GaussianTalker.
- Ablations show removing 3DMM or landmark control, or replacing AdaIN with addition or attention, degrades LMD and AVConf; full model LMD is 1.511 on Testset 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3DMM-based coarse control (ablation) | HiCoDe · HDTF | LMD: 2.187 -> 1.511 | Helped |
| add landmark-based fine control (ablation) | HiCoDe · HDTF | LMD: 1.868 -> 1.511 | Helped |
| use hierarchical control instead of directly controlling 3D Gaussians with audio features (ablation) | HiCoDe · HDTF | LMD: 5.281 -> 1.511 | Helped |
| use AdaIN instead of addition (ablation) | HiCoDe · HDTF | LMD: 2.418 -> 1.511 | Helped |
| use AdaIN instead of attention (ablation) | HiCoDe · HDTF | LMD: 2.966 -> 1.511 | Helped |
| use the proposed 3D Gaussian initialization instead of random initialization (ablation) | HiCoDe · HDTF | LMD: 1.998 -> 1.511 | Helped |
| add hierarchical facial expression control (HFEC) (ablation) | HiCoDe · HDTF | LMD: 4.258 -> 1.511 | Helped |
Surprising findings
- Wav2Lip retains the highest AVConf on HDTF (8.451 vs 8.428 for HiCoDe is close, but Wav2Lip's mouth-only editing trades off image quality for sync), and HiCoDe's AVConf is marginally lower than Wav2Lip on HDTF.
Limitations
- Evaluated on only 8 HDTF videos plus two test sets; per-identity training required.
- AVConf is marginally lower than Wav2Lip on HDTF because Wav2Lip only edits the mouth region.
- Hardware used for timing is not specified in the text.
Experimental setup
Datasets: HDTF Testset 1 Testset 2
Benchmarks: HDTF Testset 1 Testset 2
Metrics: SSIM PSNR LPIPS LMD AVConf training time FPS
Baselines: Wav2Lip PC-AVS ER-NeRF RAD-NeRF GeneFace++ GaussianTalker
Builds on: Deformable 3D Gaussians 3D Gaussian Splatting Faceformer HuBERT 3DMM
Closest papers by meaning
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.91)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.88)
- Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians (CVPR 2024 · similarity 0.87)
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.87)
- AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis (AAAI 2024 · similarity 0.87)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.