Research map / Label-efficient and robust learning / Recognition and long-tail
Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECSS, a conversational speech synthesis model that uses a heterogeneous emotional conversational graph (HGT-based) over multi-source dialogue history and a contrastive-learning emotion renderer, with new emotion and intensity labels added to DailyTalk.
Problem
Conversational speech synthesis largely ignores emotional expressiveness: emotion understanding of dialogue context and emotion rendering for the target utterance are missing, and emotional conversational speech data is scarce.
Approach
Builds a heterogeneous Emotional Conversational Graph whose nodes include text, audio, speaker, emotion and intensity, encodes it with Heterogeneous Graph Transformer, and predicts the current utterance's emotion, intensity and prosody with an emotion renderer trained with supervised contrastive losses on emotion and intensity labels.
Key results
- ECSS reaches N-DMOS 3.506 and E-DMOS 3.619, above the homogeneous graph baseline (3.384 and 3.493).
- ECSS achieves the best MAE-M (0.654) and MAE-P (0.455) among systems, with suboptimal MAE-E (0.215) and MAE-D (0.152).
- Removing the contrastive loss lowers E-DMOS from 3.619 to 3.312.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use heterogeneous emotion, intensity, speaker, and audio nodes instead of omitting emotion nodes (ablation) | ECSS · DailyTalk test | E-DMOS: 3.496 -> 3.619 | Helped |
| Use emotion-intensity nodes (ablation) | ECSS · DailyTalk test | E-DMOS: 3.523 -> 3.619 | Helped |
| Use speaker nodes (ablation) | ECSS · DailyTalk test | N-DMOS: 3.401 -> 3.506 | Helped |
| Use audio nodes (ablation) | ECSS · DailyTalk test | N-DMOS: 3.391 -> 3.506 | Helped |
| Use contrastive learning loss rather than cross-entropy loss (ablation) | ECSS · DailyTalk test | E-DMOS: 3.312 -> 3.619 | Helped |
| Increase context length from 2 to 10, then further to 14 (ablation) | ECSS · DailyTalk test | MAE-M: 0.667 -> 0.654; 0.654 -> 0.661 | Mixed |
Surprising findings
- Context length of 10 is best; both shorter (2-6) and longer (13-14) histories degrade objective metrics.
- The homogeneous graph baseline achieves the lowest MAE-E and MAE-D, slightly better than ECSS on those metrics.
Limitations
- Emotion and intensity labels were annotated by a single professional practitioner, and the emotion distribution is highly imbalanced (neutral 18,197 vs. fear 74).
- Subjective evaluation uses 30 listeners and objective metrics do not fully reflect human perception, as the authors note.
- Evaluated only on DailyTalk, a single dataset with two speakers.
Experimental setup
Datasets: DailyTalk
Benchmarks: DailyTalk
Metrics: N-DMOS E-DMOS MAE-M MAE-P MAE-E MAE-D
Baselines: FastSpeech2 without context modeling GRU-based context modeling Homogeneous graph-based context modeling
Builds on: HGT FastSpeech2 HiFi-GAN Global style tokens DialogueGCN
Closest papers by meaning
- Adaptive Graph Learning for Multimodal Conversational Emotion Detection (AAAI 2024 · similarity 0.87)
- Dual-View Learning for Conversational Emotion Recognition Through Context and Emotion-Shift Modeling (AAAI 2025 · similarity 0.86)
- Emotional Face-to-Speech (ICML 2025 · similarity 0.86)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.86)
- Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis (NeurIPS 2025 · similarity 0.86)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.