Research map / Image generation / Editing and style transfer
CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CustomListener, a text-prior-guided listening head generator that turns GPT-written listener descriptions into speaker-aligned dynamic portrait tokens and uses a past-guided diffusion model for coherent long-video generation.
Problem
Listening head generation is limited to coarse emotion labels, so users cannot freely customize a listener's identity, personality, and fine-grained motions while keeping responses synchronized with the speaker and coherent across segments.
Approach
Converts static text-derived portrait tokens into time-dependent and speaker-motion-refined dynamic tokens (SDP module) via audio-text attention, and maintains customized listener habits across segments by weighting past motions by portrait-token similarity (PGG module), which conditions a DDPM-based transformer decoder before PIRenderer rendering.
Key results
- On ViCo test set, FD 11.54 vs 14.89 for the best prior baseline (L2L), and RTLCC 6.12 vs 6.34 for L2L.
- Image quality on ViCo: SSIM 0.60, CPBD 0.19, PSNR 17.98, FID 20.06, best among compared methods.
- User study (28 raters, 1-4 scale) gave the proposed method 3.96 on similarity to GT and 3.92 on naturalness, above RLHG, PCH and L2L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Audio-text Responsive Interaction component (ablation) | CustomListener · ViCo Dtest and Dood | RTLCC: 0.206 -> 0.165 | Helped |
| Add the Speaker Motion-based Weighting component (ablation) | CustomListener · ViCo Dtest and Dood | RTLCC: 0.201 -> 0.165 | Helped |
| Add the Static to Dynamic Portrait module (ablation) | CustomListener · ViCo Dtest and Dood | RTLCC: 0.213 -> 0.165 | Helped |
| Add the Past-guided Module (ablation) | CustomListener · ViCo Dtest and Dood | FID△fm: 6.47 -> 3.42 | Helped |
Limitations
- Evaluated only on ViCo and RealTalk with 3DMM-based rendering; no body generation.
- Relies on GPT to produce text priors and on retrained baselines; ELP and MFR-Net source code unavailable so comparisons are partial (MFR-Net numbers taken from paper).
Experimental setup
Datasets: ViCo RealTalk
Benchmarks: ViCo RealTalk
Metrics: FD RTLCC RWTLCC FID_fm SND V-D SSIM CPBD PSNR FID
Baselines: RLHG PCH L2L MFR-Net
Builds on: DDPM PIRenderer Learning2Listen ELP MFR-Net TalkCLIP
Closest papers by meaning
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.86)
- INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations (CVPR 2025 · similarity 0.86)
- LLM-driven Multimodal and Multi-Identity Listening Head Generation (CVPR 2025 · similarity 0.86)
- GLARE: Generating Listening Heads with Appropriate REactions (NeurIPS 2026 · similarity 0.85)
- Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (NeurIPS 2025 · similarity 0.84)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.