Research map / Image generation / Autoregressive and latent generation
EmoReg: Directional Latent Vector Modeling for Emotional Intensity Regularization in Diffusion-based Voice Conversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EmoReg, a diffusion-based emotional voice conversion model that regulates emotion intensity by shifting SSL (emotion2vec) emotion embeddings along PCA-reduced GMM-based directional vectors between emotion states.
Problem
Emotional voice conversion lacks fine-grained emotion intensity control for discrete-emotion systems, since intensity labels or naive embedding interpolation degrade quality and labeled intensity data is scarce.
Approach
Models emotion-transition directions in an SSL emotion embedding space (GMM local means, pairwise neutral-to-emotion differences, PCA to 128 components) and scales the resulting direction by an intensity value before conditioning a DiffVC-style SDE decoder.
Key results
- Proposed EmoReg achieves average emotion similarity 0.96 on English versus 0.94 for the best baseline (EmoVox) and 0.94 for the ablation without DVM.
- Proposed approach yields 13.56% absolute MOS gain over EmoVox and 43.13% over Mixed Emotion on speech quality; 5.52% over the no-DVM ablation.
- Average WER on English improves to 13.69 for proposed vs 13.88 for ablation and 36.23 for EmoVox; 128 PCA components chosen as best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DVM to EmoReg (ablation) | EmoReg · ESD English, average emotion similarity | emotion similarity: 0.94 ± 0.004 -> 0.96 ± 0.003 | Helped |
| add DVM to EmoReg (ablation) | EmoReg · English and Hindi, Neu-Ang, Neu-Sad, and Neu-Hap | emotion similarity: English average: 0.94 ± 0.004 -> 0.96 ± 0.003; Hindi average: 0.88 -> 0.88 | Mixed |
| add DVM to EmoReg (ablation) | EmoReg · English, average across intensity scales | WER: 13.88 -> 13.69 | Helped |
| add DVM to EmoReg (ablation) | EmoReg · English, average across intensity scales | CER: 8.12 -> 8.07 | Helped |
| add DVM to EmoReg (ablation) | EmoReg · English, speech quality | MOS: +5.52% | Helped |
| add DVM to EmoReg (ablation) | EmoReg · Hindi | WER: 33.33 -> 32.93 | Helped |
| add DVM to EmoReg (ablation) | EmoReg · Hindi | CER: 17.74 -> 18.20 | Hurt |
| use 128 instead of 64 or 256 PCA components (ablation) | EmoReg DVM · English, Neu-Hap | emotion similarity: 0.911 -> 0.934; 0.928 -> 0.934 | Helped |
| use 128 instead of 64 or 256 PCA components (ablation) | EmoReg DVM · Hindi, Neu-Ang and Neu-Hap | emotion similarity: Neu-Ang: 0.889 -> 0.898; Neu-Hap: 0.849 -> 0.858 | Helped |
| increase emotion intensity scale | EmoReg with DVM · Neutral-to-emotional conversion | emotion similarity | Helped |
| use DVM rather than interpolation-based control (ablation) | EmoReg · Neutral-to-emotional conversion, increasing intensity | emotion similarity | Helped |
| use DVM rather than interpolation-based control (ablation) | EmoReg · English, intensity scales 0.0, 0.2, and 0.4 | WER: 13.86 vs 14.6; 13.34 vs 13.65; 13.75 vs 13.89 | Hurt |
Surprising findings
- The ablation without DVM achieves lower WER than the full model at low intensity scales (0.0 to 0.4), which the authors attribute to traversing closely related emotional spaces.
Limitations
- Intensity-controllability comparison restricted to EmoVox and Mixed Emotion since other baselines lack intensity support.
- Internal Hindi dataset is not publicly available, limiting reproducibility.
- Method has been evaluated only on clean speech; background music or noise is left for future work.
Experimental setup
Datasets: ESD internal Hindi emotional speech database
Benchmarks: ESD
Metrics: emotion similarity (cosine) WER CER MOS AutoPCP
Baselines: EmoVox Mixed Emotion CycleGAN-EVC StarGAN-EVC StyleVC DISSC Seq2Seq-EVC EmoReg w/o DVM
Builds on: DiffVC emotion2vec Montreal Forced Aligner HiFiGAN
Closest papers by meaning
- Emotional Face-to-Speech (ICML 2025 · similarity 0.84)
- Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis (NeurIPS 2025 · similarity 0.84)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.84)
- Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling (AAAI 2024 · similarity 0.84)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.83)
- EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.