Research map / Vision-language and multimodal / CLIP and zero-shot

Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis

NeurIPS 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Language models for zero-shot reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WeSCon, a two-stage self-training framework that gives a pretrained zero-shot TTS model (CosyVoice2) word-level emotion and speaking-rate control via multi-round teacher inference, then distills it into an end-to-end student with a dynamic emotional attention bias.

Problem

Existing emotional and zero-shot TTS controls operate at utterance level, and word-level emotion and speaking-rate control requires intra-sentence emotion-transition data that is scarce.

Approach

A teacher uses multi-round inference with tail-to-head transition smoothing, a content aligner, and token resampling for speed control to produce emotion-transition speech from GPT-4o-generated scripts; the student TTS is fine-tuned on filtered teacher outputs with a dynamic emotional attention bias and emotion flags, enabling single-pass word-level control.

Key results

• WeSCon (2nd stage) reaches DNSV 4.361 (English) and 4.210 (Chinese) versus 7.894 and 7.612 for CosyVoice2, with the highest Emo2v. (0.882 EN, 0.872 ZH) among compared systems.
• In subjective MOS with 15 listeners, WeSCon scores EMOS 3.70, SPMOS 3.89, SMOS 3.96, NMOS 3.93, the highest of all compared systems.
• On standard zero-shot TTS (SEED test-zh), the 2nd-stage student gets CER 1.47 and S-SIM 0.744, close to CosyVoice2 (1.45, 0.748).

Measured effects reported in the paper

Surprising findings

• Removing speed control only slightly changes CER (2.126 vs 2.129), while removing the attention bias or data formatting causes larger drops, notably CER rising to 4.141 without the naive-to-pretraining data format.
• Self-training data scaling peaks at about 500 hours and metrics decline beyond that, attributed to limited emotion and speaker diversity in ESD.

Limitations

• Emotion transitions are predefined by GPT-4o-generated plans and cannot adapt to context, limiting flexibility.
• Supports only a fixed set of discrete emotions without blended or compositional expressions.
• Lacks semantic modeling of gradual emotional evolution between states.

Experimental setup

Datasets: ESD LibriSpeech-100-Clean AISHELL-1 CASIA SEED test-zh

Benchmarks: ESD test set (constructed word-level emotion-speed test sets) CASIA (out-of-domain) SEED test-zh

Metrics: WER/CER DNSV speaker similarity (WavLM-Large cosine) AutoPCP emotion2vec similarity arousal similarity MOS (EMOS, SPMOS, SMOS, NMOS)

Baselines: Index-TTS F5-TTS Spark-TTS CosyVoice2

Builds on: CosyVoice2 flow matching

Closest papers by meaning

• Emotional Face-to-Speech (ICML 2025 · similarity 0.86)
• Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.86)
• Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling (AAAI 2024 · similarity 0.86)
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.84)
• EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.84)
• EmoReg: Directional Latent Vector Modeling for Emotional Intensity Regularization in Diffusion-based Voice Conversion (AAAI 2025 · similarity 0.84)