Research map / Label-efficient and robust learning / Self-supervised and contrastive
HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis
Sang-Hoon Lee, Seung-Bin Kim, Ji-Hyun Lee, Eunwoo Song, Min-Jae Hwang, Seong-Whan Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use linear spectrogram instead of 1st layer of XLS-R as speaker encoder input (ablation) | HierSpeech · speaker encoder input | EER: 4.62 -> 2.77 | Improved |
| use 12th layer of XLS-R instead of 1st layer as speaker encoder input (ablation) | HierSpeech · speaker encoder input | EER: 4.62 -> 10.23 | Hurt |
| use 12th layer of XLS-R for the linguistic encoder (zl) rather than 1st layer (ablation) | HierSpeech · frame-level speaker classification | speaker accuracy of zl: 15.09% -> 8.95% | Improved |
| increase VITS flow blocks from 4 to 8 (ablation) | VITS · VCTK ablation | PER: 12.24 -> 13.42 | Hurt |
| add phoneme predictor (PP) to VITS posterior encoder (ablation) | VITS · VCTK ablation | PER: 12.24 -> 7.60 | Improved |
| add phoneme predictor (PP) to VITS posterior encoder (ablation) | VITS · VCTK ablation | WER: 30.62 -> 22.98 | Improved |
| add acoustic encoder (AE) between linguistic representation and waveform decoder (ablation) | HierSpeech · VCTK ablation | PER: 8.00 -> 6.25 | Improved |
| add both AE and PP to the model (ablation) | HierSpeech · VCTK ablation | PER: 16.70 -> 6.25 | Improved |
| replace VITS with HierSpeech hierarchical variational inference (ablation) | VITS · VCTK TTS | WER: 25.54 -> 19.55 | Improved |
| replace VITS with HierSpeech (ablation) | VITS · LibriTTS speaker transfer | MCD: 4.37 -> 4.42 | Hurt |
| replace VITS with HierSpeech (ablation) | VITS · LibriTTS speaker transfer | PER: 13.62 -> 7.47 | Improved |
| fine-tune without transcripts (HierSpeech-U) instead of with transcripts (ablation) | HierSpeech · few-shot speaker adaptation, VCTK | WER: 16.95 -> 15.85 | Improved |
Closest papers by meaning
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (ICML 2021 · similarity 0.87)
- Bidirectional Variational Inference for Non-Autoregressive Text-to-Speech (ICLR 2021 · similarity 0.87)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.86)
- A Variational Framework for Improving Naturalness in Generative Spoken Language Models (ICML 2025 · similarity 0.85)
- High-Fidelity Simultaneous Speech-To-Speech Translation (ICML 2025 · similarity 0.85)
- HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
