Research map / Vision-language and multimodal / Cross-modal learning
Expressive Talking Head Generation With Granular Audio-Visual Control
Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, Jingdong Wang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the granular audio-driven pose, expression, and speech-content controls | GC-AVT · MEAD | LMDm: 4.014 -> 3.252 | Improved |
| Add the granular audio-driven pose, expression, and speech-content controls | GC-AVT · VoxCeleb2 | LMDm: 3.662 -> 3.356 | Improved |
| Add independent pose, expression, and visual speech-content control | GC-AVT · MEAD | LMDm: 2.966 -> 2.349 | Improved |
| Add independent pose, expression, and visual speech-content control | GC-AVT · VoxCeleb2 | LMDm: 4.035 -> 2.811 | Improved |
| Add VGG and VGGFace perceptual losses (ablation) | GC-AVT · VoxCeleb2 | LMD: 4.753 -> 3.025 | Improved |
| Add VGG and VGGFace perceptual losses (ablation) | GC-AVT · VoxCeleb2 | SSIM: 0.662 -> 0.710 | Improved |
| Add contrastive audio-visual synchronization loss (ablation) | GC-AVT · VoxCeleb2 | Syncconf: 4.066 -> 5.250 | Improved |
| Add contrastive audio-visual synchronization loss (ablation) | GC-AVT · VoxCeleb2 | LMDm: 4.311 -> 3.356 | Improved |
| Add the time-shifted mouth input to expression processing (ablation) | GC-AVT · VoxCeleb2 | LMDm: 3.704 -> 3.356 | Improved |
| Add the time-shifted mouth input to expression processing (ablation) | GC-AVT · VoxCeleb2 | Syncconf: 4.760 -> 5.250 | Improved |
| Remove time-shifted mouth input from expression processing (ablation) | GC-AVT · VoxCeleb2 | SSIM: 0.684 -> 0.710 | No clear effect |
| Use the proposed masking design rather than the tested alternative mask layouts (ablation) | GC-AVT · VoxCeleb2 | speech-content control | Improved |
Closest papers by meaning
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.90)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.90)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.89)
- SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · similarity 0.89)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
