Research map / Video / Video generation
GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GoHD, an audio-driven talking-head framework combining a latent-navigation face animator with gaze control, a conformer-based conditional diffusion model for prosody-aware head poses, and a two-stage (lip distillation then LSTM eye-motion) expression predictor.
Problem
Audio-driven talking head generation struggles with generalizing to diverse unseen portraits, disentangling identity and motion, natural gaze, prosody-aware yet diverse head poses, and lip-synchronized expressions with limited training data.
Approach
Latent-navigation animator that predicts motion as a linear combination of learned orthogonal motion directions added to the identity latent, with gaze as a driving condition; a conformer-denoised diffusion model for poses; and a two-stage expression predictor that distills lip motion from a lip-sync expert and then generates eye motion with an LSTM.
Key results
- On HDTF, GoHD achieves FID 30.40 vs. 36.91 for SadTalker and 34.28 for MakeItTalk, and on VoxCeleb FID 58.11 vs. 60.75 for SadTalker.
- User study with 37 volunteers over 30 examples: GoHD receives 70.99% of responses for overall naturalness.
- Ablation: removing the two-stage strategy (w/o Stage 2 & eye features) gives the best MLD (1.785) but much lower eye SSIM (0.813) than the full LSTM variant (0.915).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Stage 2 eye-motion generation and Stage 1 eye-motion features (ablation) | GoHD expression predictor · HDTF | SSIM_e: 0.813 -> 0.915 | Helped |
| Add Stage 1 pretraining (ablation) | GoHD expression predictor · HDTF | MLD: 1.931 -> 1.792 | Helped |
| Use lip-expert distillation instead of ground-truth lip motions as Stage 1 targets (ablation) | GoHD expression predictor · HDTF | MLD: 2.012 -> 1.792 | Helped |
| Use an LSTM rather than a transformer for eye-motion generation (ablation) | GoHD expression predictor · HDTF | SSIM_e: 0.894 -> 0.915 | Helped |
| Add the parsing loss (ablation) | GoHD face animator · HDTF | LSE-C: 7.96 -> 8.13 | Helped |
| Add the lip-and-eye partial mask (ablation) | GoHD face animator · HDTF | LSE-D: 7.77 -> 7.78 | Mixed |
| Add lip-reading loss (ablation) | GoHD Stage 1 expression predictor · HDTF | MLD: 1.76 -> 1.65 | Helped |
| Add landmark fidelity loss (ablation) | GoHD Stage 1 expression predictor · HDTF | MLD: 1.79 -> 1.65 | Helped |
| Add shut-up loss (ablation) | GoHD Stage 1 expression predictor · HDTF | MLD: 1.72 -> 1.65 | Helped |
| Add structural similarity loss in Stage 2 (ablation) | GoHD expression predictor · HDTF | SSIM_e: 0.845 -> 0.915 | Helped |
| Add the eye-motion discriminator (ablation) | GoHD expression predictor · HDTF | SSIM_e: 0.840 -> 0.915 | Helped |
| Add the temporal coefficient-sequence discriminator (ablation) | GoHD expression predictor · HDTF | SSIM_e: 0.844 -> 0.915 | Helped |
Surprising findings
- Wav2Lip gets better LSE scores than ground truth because it is trained with the same SyncNet used for evaluation, so the authors discount SyncNet metrics.
- The variant without Stage 2 achieves the lowest mouth landmark distance, indicating a trade-off between lip accuracy and eye naturalness.
Limitations
- Explicit 3DMM-based motion descriptors cannot capture subtle head or facial movements, causing loss of detail such as wrinkles and teeth.
- Frame-by-frame animation ignores temporal continuity, causing slight jitter.
- Relies on SyncNet-based metrics that are not definitive for lip sync.
Experimental setup
Datasets: VoxCeleb HDTF
Benchmarks: HDTF VoxCeleb
Metrics: LSE-C LSE-D FID SSIM Var CSIM MLD
Baselines: MakeItTalk Wav2Lip Audio2Head EAMM SadTalker
Builds on: LIA DDPM SadTalker classifier-free guidance
Closest papers by meaning
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.91)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.90)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.90)
- SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · similarity 0.90)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.90)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.