Research map / Video / Video generation
MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoDiTalker, a two-stage motion-disentangled diffusion framework that first generates 3D facial landmarks from audio (AToM) and then renders talking head video with a tri-plane-conditioned video diffusion model (MToV).
Problem
Audio-driven talking head generation suffers from GAN training instability and mode collapse, while prior diffusion-based methods struggle with weak lip-audio alignment, temporal inconsistency from frame-by-frame generation, and slow sampling.
Approach
Disentangle generation into audio-to-motion (a transformer diffusion model predicting residual 3D landmarks with separate lip-related and lip-unrelated attention branches) and motion-to-video (a latent video diffusion model conditioned on tri-plane encodings of landmarks, pose frames, and identity frames), so that video synthesis is driven by explicit motion rather than audio embeddings in cross-a
Key results
- On HDTF, MoDiTalker reports FID 14.15 and PSNR 35.82 versus DreamTalk's FID 124.51 and PSNR 29.59, and LMD 1.38 versus 2.56.
- MoDiTalker's LSE-D is 9.15, worse than Wav2Lip (5.86) and DreamTalk (8.37), which the authors attribute to not using SyncNet loss.
- Generating a 5-second 25 fps video takes 23 seconds, reported as 43x faster than DiffTalk and 31x faster than Diffused Heads.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoDiTalker instead of Wav2Lip | HDTF talking-head generation · HDTF test | LSE-D: 5.86 -> 9.15 | Hurt |
| Add residual prediction to the audio-only AToM baseline (ablation) | AToM · HDTF test | LSE-C: 0.316 -> 0.386 | Helped |
| Add initial landmark embedding F_L to residual AToM (ablation) | AToM · HDTF test | LMD: 1.47 -> 1.31 | Helped |
| Add disentangled attention to AToM (ablation) | AToM · HDTF test | LMD: 1.31 -> 1.26 | Helped |
| Add disentangled attention to AToM (ablation) | AToM · HDTF test | LSE-C: 0.385 -> 0.390 | Helped |
| Add identity frames X_I (ablation) | MToV · HDTF test | CSIM | Helped |
| Add pose frames X_P to MToV with identity frames (ablation) | MToV · HDTF test | CSIM: 0.87 -> 0.91 | Helped |
| Add pose frames X_P to MToV with identity frames (ablation) | MToV · HDTF test | LMD: 11.18 -> 5.23 | Helped |
| Add audio embedding A through cross-attention instead of using landmarks (ablation) | MToV · HDTF test | LMD: 1.38 -> 2.67 | Hurt |
| Add facial motion landmarks L instead of audio embedding A (ablation) | MToV · HDTF test | LMD: 2.67 -> 1.38 | Helped |
Surprising findings
- Despite strong FID, PSNR and LMD, MoDiTalker's LSE-D (9.15) is worse than several baselines, including Wav2Lip and DreamTalk.
- The authors could not reproduce DiffTalk's reported results and therefore excluded it from quantitative comparison.
Limitations
- Temporal consistency between generated frames is sometimes less ideal and may need post-processing.
- HDTF lacks dynamic poses, limiting generation of diverse head motion.
- Lip-sync score (LSE-D) is weaker than some baselines since SyncNet loss is not used in training.
Experimental setup
Datasets: LRS3-TED HDTF
Benchmarks: HDTF CREMA-D VoxCeleb2
Metrics: FID CPBD PSNR LPIPS CSIM LMD LSE-D LSE-C
Baselines: Wav2Lip PC-AVS MakeItTalk Audio2Head DiffusedHead DreamTalk GeneFace
Builds on: latent diffusion models GeneFace DreamTalk HuBERT Deep 3D Reconstruction
Closest papers by meaning
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.90)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.90)
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking (NeurIPS 2025 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.89)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.