Research map / Label-efficient and robust learning / Recognition and long-tail
Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the synthesizer on a larger and more diverse speaker set | Multispeaker TTS · Unseen speakers, VCTK and LibriSpeech | speaker similarity: VCTK: 10.46% -> 6.26% SV-EER; LibriSpeech: 29.19% -> 5.08% SV-EER | Helped |
| Train the speaker encoder on more speakers, scaling from 1.2K to 18K (ablation) | Proposed multispeaker TTS · LibriSpeech Clean held-out speakers | speaker similarity MOS: $2.23 \pm 0.08$ -> $3.03 \pm 0.09$ | Helped |
| Train the speaker encoder on more speakers, scaling from 1.2K to 18K (ablation) | Proposed multispeaker TTS · LibriSpeech Clean held-out speakers | SV-EER: 16.60% -> 5.08% | Helped |
| Replace LS-Clean encoder data with LS-Other data at a similar speaker count (ablation) | Proposed multispeaker TTS · LibriSpeech Clean held-out speakers | speaker similarity MOS: $2.23 \pm 0.08$ -> $2.27 \pm 0.09$ | No clear effect |
| Add VoxCeleb data to LS-Other encoder training (ablation) | Proposed multispeaker TTS · LibriSpeech Clean held-out speakers | speaker similarity MOS: $2.27 \pm 0.09$ -> $2.43 \pm 0.09$ | Helped |
| Add VoxCeleb2 data to LS-Other + VoxCeleb encoder training (ablation) | Proposed multispeaker TTS · LibriSpeech Clean held-out speakers | speaker similarity MOS: $2.43 \pm 0.09$ -> $2.54 \pm 0.09$ | Helped |
| Increase reference audio from 1 sec to 3 sec (ablation) | Proposed multispeaker TTS · VCTK | speaker similarity MOS: $2.85 \pm 0.07$ -> $3.31 \pm 0.07$ | Helped |
| Increase reference audio from 3 sec to 10 sec (ablation) | Proposed multispeaker TTS · VCTK | speaker similarity MOS: $3.31 \pm 0.07$ -> $3.18 \pm 0.07$ | Hurt |
| Use 1 sec rather than 5 sec of reference audio (ablation) | Proposed multispeaker TTS · VCTK | naturalness MOS: $4.20 \pm 0.06$ -> $4.28 \pm 0.05$ | Helped |
| Add a speaker discrimination loss to jointly trained speaker encoder and synthesizer (ablation) | Jointly trained model · LibriSpeech Unseen | speaker similarity MOS: 2.44 ± 0.09 -> 2.12 ± 0.08 | Hurt |
| Use separate encoder and synthesizer training rather than joint training with speaker loss (ablation) | Multispeaker TTS · LibriSpeech Unseen | naturalness MOS: 3.71 ± 0.06 -> 4.12 ± 0.05 | Helped |
Closest papers by meaning
- Deep Voice 2: Multi-Speaker Neural Text-to-Speech (NeurIPS 2017 · similarity 0.91)
- ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis (ICML 2024 · similarity 0.88)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
- SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICML 2024 · similarity 0.87)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.87)
- DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.