Research map / Large language models / Language, safety and interpretability

LLM-driven Multimodal and Multi-Identity Listening Head Generation

CVPR 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends LLM-based listener head generation (LM-listener) with multimodal speaker tokens (speech content, acoustics, emotion) and an identity-disentangled VQ-VAE using instance and adaptive instance normalization for flexible identity control.

Problem

LLM-based listening head generation relies only on speech content and entangles listener identity with facial motion generation, limiting naturalness, fidelity, and generalization across listener identities.

Approach

Feed the LLM interleaved multimodal speaker tokens (Whisper text, SpeechTokenizer acoustic tokens, EMOCA emotion tokens) and decouple identity by using instance normalization in the VQ-VAE encoder to erase identity and AdaIN in the decoder to inject 3DMM identity parameters.

Key results

• On L2L-trevor, Ours (GPT2) reduces L2 from 0.4485 (LM-listener GPT2) to 0.2848 and FD from 18.52 to 9.81, roughly 38% and 45% improvements.
• On RealTalk, Ours improves L2 from 0.2026 to 0.0860 and FD from 9.60 to 3.39 over LM-listener.
• In a 33-person user study, volunteers preferred the method over LM-listener in 86.4% of instances and over ground truth in 65.8%.

Measured effects reported in the paper

Surprising findings

• Adding implicit speech content tokens from SpeechTokenizer's first quantizer gave no gains over text alone, attributed to semantic redundancy with Whisper text.
• Speech acoustic tokens alone gave limited improvement, but combined with emotion tokens they yielded the best results.

Limitations

• LLaMA2-7B gave no clear gain over GPT2-Medium, attributed to limited dataset size and fine-tuning variations.
• Evaluated only on two listener datasets (single-listener L2L-trevor and multi-listener RealTalk).

Experimental setup

Datasets: L2L-trevor RealTalk

Benchmarks: L2L-trevor RealTalk

Metrics: L2 FD Variation Diversity P-FD L2 Affect user preference

Baselines: Random NN Facial Motion NN Speech Content Naive Random Walk LM-listener (GPT2) LM-listener (LLAMA2)

Builds on: LM-listener Learning2Listen VQ-VAE SpeechTokenizer EMOCA

Closest papers by meaning

• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (AAAI 2024 · similarity 0.86)
• FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio (CVPR 2024 · similarity 0.86)
• OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions (NeurIPS 2025 · similarity 0.86)
• DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations (CVPR 2025 · similarity 0.86)
• INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations (CVPR 2025 · similarity 0.86)
• CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation (CVPR 2024 · similarity 0.86)