Research map / Video / Video generation
Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCDM, a motion-prior conditional diffusion model for long-term TalkingFace video generation using archived-clip and present-clip motion priors plus memory-efficient temporal attention, and releases the TalkingFace-Wild multilingual dataset.
Problem
Long-term TalkingFace generation suffers from inconsistent head movement, expression, and lip sync, plus identity drift and error accumulation over extended sequences in diffusion models.
Approach
Conditions a SD v1.5-style denoising UNet on long archived-clip history (via frame-aligned attention over frozen VAE tokens), predicts head/lip/expression motion tokens with a present-clip motion-prior diffusion transformer, and uses a memory-update temporal attention to accumulate motion features and reduce error accumulation.
Key results
- On HDTF, MCDM reports FID 26.45 and FVD 543.28 versus 31.37 and 586.10 for the strongest baseline MegActor-Σ.
- On TalkingFace-Wild, MCDM achieves FID 42.08 and FVD 656.71, the best among compared methods (MegActor-Σ 48.57 / 724.40).
- Ablations show removing archived-clip history (w/o F_a) raises FID from 42.08 to 46.25 on TalkingFace-Wild.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add archived historical frames (F_a) (ablation) | MCDM · TalkingFace-Wild | FID: 46.25 -> 42.08 | Helped |
| Add the present-clip motion-prior instead of directly using audio attention (ablation) | MCDM · TalkingFace-Wild | FID: 45.63 -> 42.08 | Helped |
| Use memory-efficient temporal attention instead of standard temporal attention (ablation) | MCDM · TalkingFace-Wild | FID: 44.27 -> 42.08 | Helped |
| Replace frame-aligned attention with Q-Former (ablation) | MCDM · TalkingFace-Wild | FID: 42.08 -> 42.49 | Hurt |
| Replace the frozen VAE and trainable patchify layer with Reference UNet, omitting archived-clip information (ablation) | MCDM · TalkingFace-Wild | FID: 42.08 -> 47.12 | Hurt |
| Increase the memory update α to 0.9 (ablation) | MCDM · TalkingFace-Wild | SSIM | Hurt |
| Use MCDM rather than MegActor-Σ | TalkingFace-Wild · TalkingFace-Wild | FID: 48.57 -> 42.08 | Helped |
Limitations
- Ablation and design choices (alpha, landmark dropout) reported only on TalkingFace-Wild; the paper's own tables contain inconsistencies in baseline numbers across datasets.
- Lip and expression encoders are custom-trained and details are deferred to supplementary material, limiting reproducibility.
- Code, models and datasets are only promised 'will be publicly available' in the abstract.
Experimental setup
Datasets: HDTF CelebV-HQ TalkingFace-Wild
Benchmarks: HDTF CelebV-HQ TalkingFace-Wild
Metrics: FID FVD Sync-C Sync-D SSIM E-FID
Baselines: Audio2Head V-Express AniPortrait SadTalker Hallo EchoMimic MegActor-Σ
Builds on: Stable Diffusion v1.5 AnimateDiff Wav2Vec CLIP FiLM
Closest papers by meaning
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.90)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.89)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.89)
- MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation (AAAI 2025 · similarity 0.88)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.