Research map / Large language models / Language, safety and interpretability

Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing

CVPR 2025 · Large language models › Language, safety and interpretability · approach: Sign and cross-lingual translation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a two-stage movie dubbing method: prosody-enhanced acoustic pre-training on LibriTTS, then a frozen-acoustic prosody adapting stage with a prosodic text BERT encoder, prosodic style diffusion, and in-domain emotion analysis for video-synchronized dubbing.

Problem

Movie dubbing (visual voice cloning) must generate speech from scripts that is temporally and emotionally aligned with a silent movie clip while keeping high acoustic quality, but limited, noisy dubbing datasets degrade acoustic modeling.

Approach

Pre-train the entire acoustic system on a large clean TTS corpus with pitch-shifting and duration-stretching prosody augmentation, then freeze it and train only disentangled prosody modules (prosodic text BERT, prosodic style diffusion, visual emotion features with in-domain normalization) on the dubbing data.

Key results

• On V2C-Animation Dub 2.0, WER drops to 11.50% (vs 17.33% for Speaker2Dubber) and UT-MOS rises to 3.09 (vs 2.42).
• EMO-ACC of 48.93% on V2C-Animation Dub 1.0, a +2.58 point gain over the best prior method.
• Zero-shot test on GRID reference audio yields WER 9.85% and UT-MOS 3.63 versus 16.05% and 3.08 for Speaker2Dubber.

Measured effects reported in the paper

Surprising findings

• Ground-truth-mel plus vocoder resynthesis scores lower UT-MOS (2.18) than the proposed method's output (3.10) on V2C-Animation, suggesting the pre-trained clean-corpus decoder improves perceived quality beyond the noisy dubbing data.

Limitations

• SECS voice similarity is lower than StyleDubber on V2C-Animation Dub 1.0 (75.46 vs 82.26)
• Acoustic quality is limited by noisy movie background in dubbing datasets
• Evaluation restricted to English datasets (V2C-Animation, GRID)

Experimental setup

Datasets: LibriTTS-460 V2C-Animation GRID

Benchmarks: V2C-Animation GRID

Metrics: SECS WER UT-MOS EMO-ACC MCD-DTW MCD-DTW-SL MOS-N MOS-S CMOS

Baselines: StyleSpeech Zero-shot TTS V2C-Net HPMDubbing Face-TTS StyleDubber Speaker2Dubber

Builds on: StyleTTS HPMDubbing Speaker2Dubber StyleDubber HiFi-GAN

Closest papers by meaning

• EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing (CVPR 2025 · similarity 0.90)
• AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency (NeurIPS 2026 · similarity 0.86)
• ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.86)
• ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis (AAAI 2025 · similarity 0.85)
• EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.84)
• Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.84)