Research map / Video / Video generation
EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EmoDubber, a movie dubbing architecture that combines duration-level contrastive lip-prosody alignment, phoneme enhancement via conformer, and flow-matching synthesis with positive/negative classifier guidance to give users controllable emotion type and intensity.
Problem
Visual voice cloning (movie dubbing) struggles to jointly achieve audio-visual lip sync and clear pronunciation, and existing dubbing models cannot express user-defined emotions with controllable intensity.
Approach
Duration-level contrastive learning aligns lip motion with phoneme prosody, video-level phoneme expansion with a conformer and CTC improves pronunciation, and flow-matching synthesis uses positive and negative classifier guidance (PNGM) to amplify the chosen emotion while suppressing others.
Key results
- On Chem (setting 1.0), EmoDubber achieves LSE-C 8.11 and LSE-D 6.92 versus StyleDubber's 3.87 and 10.92, and WER 11.72 versus 13.14.
- On GRID setting 2.0, LSE-C improves by 19.46% and LSE-D by 24.12% over the best baselines, with WER 18.53% below ground truth's 22.41%.
- Emotion control with FUEC yields Intensity Score ranging from 0 to higher values, with the baseline capped at a narrower range; emotional speech WER stays below 12% for six of seven emotions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PE (ablation) | EmoDubber · Chem, Dubbing Setting 2.0 | WER: 53.36 -> 12.81 | Helped |
| add SIA (ablation) | EmoDubber · Chem, Dubbing Setting 2.0 | SECS: 82.04 -> 85.06 | Helped |
| add LPA (ablation) | EmoDubber · Chem, Dubbing Setting 2.0 | LSE-C: 4.47 -> 8.09 | Helped |
| use EmoDubber instead of StyleDubber | Chem dubbing · Chem, Dubbing Setting 2.0 | MCD: 6.01 -> 6.51 | Hurt |
| use EmoDubber instead of StyleDubber | GRID dubbing · GRID, Dubbing Setting 2.0 | WER: 19.58 -> 19.75 | Hurt |
| use EmoDubber instead of Speak2Dub | GRID dubbing · GRID, Dubbing Setting 2.0 | WER: 17.42 -> 19.75 | Hurt |
| increase positive guidance scale α | EmoDubber FUEC · Chem emotion control | Intensity Score | Helped |
| add positive and negative guidance for emotion control | EmoDubber FUEC · Chem emotion control | Intensity Score | Helped |
Surprising findings
- Removing the lip-prosody alignment module (w/o LPA) sharply degrades lip sync (LSE-C 4.47 vs 8.09 on Chem), while removing PE raises WER from 12.81 to 53.36.
Limitations
- Sad emotion produces blurred pronunciation with WER above 12% (12.12%)
- Zero-shot speaker results are reported only on GRID with Chem reference audio and a small subjective study
- Code and trained models are promised ('will be made available') rather than confirmed released
Experimental setup
Datasets: Chem GRID V2C LibriSpeech Emobox
Benchmarks: Chem GRID V2C-Animation
Metrics: LSE-C LSE-D WER SECS MCD MCD-SL MOS-S MOS-N Intensity Score
Baselines: Fastspeech2 StyleSpeech Face-TTS V2C-Net HPMDubbing Speaker2Dub StyleDubber GenerSpeech Zero-shot TTS
Builds on: Matcha-TTS StyleDubber HPMDubbing classifier guidance monotonic alignment search
Closest papers by meaning
- Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing (CVPR 2025 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.85)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.85)
- Emotional Face-to-Speech (ICML 2025 · similarity 0.85)
- Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.85)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.