Research map / Video / Video generation
Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a diffusion-based listening head generator that combines a lightweight diffusion transformer for explicit 3DMM-style motion, pose/expression-specific control, and an implicit refinement module within an SD 1.5 rendering backbone to produce 512x512 listener videos.
Problem
Listening head generation requires realistic, expressive non-verbal listener motion and high-fidelity video, but prior two-stage methods use deterministic rendering at low resolution (e.g. 256x256), and diffusion video models need far more data than the few hours of listening head data available.
Approach
A hybrid motion modeling scheme: an explicit diffusion motion generator (MediaPipe pose and expression coefficients) guides an LDM renderer, with separate control paths for head pose (pose images) and expressions (cross-attention), plus an implicit cross-attention refinement module that restores facial details lost in explicit coefficients.
Key results
- On RealTalk, SSIM 0.65 vs 0.57 for the best baseline (RLHG/PCH) and FID 13.38 vs 20.05 (PCH), a 33% relative FID improvement.
- On ViCo, the method achieves SSIM 0.67 and PSNR 20.60, and the best motion quality on the OOD test set (FID∆fm 5.05 exp / 2.06 pose vs CustomListener 5.94 / 3.59 is not directly comparable; best on OOD per authors).
- Ablations show removing explicit motion guidance degrades FID∆fm, and concatenating pose with expression instead of separate control increases expression FD from 7.48 to 7.81.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hybrid explicit motion guidance and implicit motion refinement (ablation) | listening head generation · RealTalk | SSIM: 0.63 -> 0.65 | Helped |
| add explicit motion guidance (ablation) | listening head generation · RealTalk | FID∆fm expression: 0.12 -> 0.05 | Helped |
| add implicit motion refinement (ablation) | listening head generation · RealTalk | SSIM: 0.64 -> 0.65 | Helped |
| add implicit motion refinement (ablation) | listening head generation · RealTalk | FD expression: 11.00 -> 10.48 | Helped |
| use separate pose-image and expression-feature controls instead of concatenating pose and expression (ablation) | listening head generation · RealTalk | FD expression: 7.81 -> 7.48 | Helped |
| use hybrid diffusion motion generation and high-fidelity rendering instead of prior listening-head methods | listening head generation · RealTalk | SSIM: 0.57 -> 0.65 | Helped |
| use hybrid diffusion motion generation and high-fidelity rendering instead of prior listening-head methods | listening head generation · ViCo OOD | FD expression: 12.67 -> 11.89 | Helped |
Surprising findings
- CustomListener, which uses ground-truth motion text descriptions, yields more similar motion to ground truth on the intense ViCo test set than the proposed method.
Limitations
- Conditioning restricted to speaker audio and facial expressions; no text or motion-style control.
- Rendering with the LDM backbone is computationally expensive, the main speed bottleneck.
- On the intense ViCo test set, the method underperforms CustomListener on motion similarity, which uses extra ground-truth text descriptions.
Experimental setup
Datasets: ViCo RealTalk VFHQ
Benchmarks: ViCo RealTalk
Metrics: SSIM PSNR FID FD FID∆fm SND
Baselines: RLHG PCH L2L MFR-Net CustomListener
Builds on: Stable Diffusion 1.5 AnimateDiff EMO PIRenderer DDPM
Closest papers by meaning
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.91)
- MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation (AAAI 2025 · similarity 0.90)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- 🎧MOSPA: Human Motion Generation Driven by Spatial Audio (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.