Research map / Label-efficient and robust learning / Recognition and long-tail
Neural Voice Cloning with a Few Samples
Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, Yanqi Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use whole-model adaptation instead of embedding-only adaptation (ablation) | Speaker adaptation · VCTK (108 speakers; 10 cloning samples) | MOS: 2.67$\pm$0.10 -> 3.16$\pm$0.09 | Improved |
| Use whole-model adaptation instead of embedding-only adaptation (ablation) | Speaker adaptation · VCTK (108 speakers; 10 cloning samples) | Similarity score: 2.95$\pm$0.09 -> 3.16$\pm$0.08 | Improved |
| Add joint fine-tuning of the encoder and generative model (ablation) | Speaker encoding · VCTK (108 speakers; 10 cloning samples) | MOS: 2.79$\pm$0.10 -> 2.99$\pm$0.12 | Improved |
| Add joint fine-tuning of the encoder and generative model (ablation) | Speaker encoding · VCTK (108 speakers; 10 cloning samples) | Similarity score: 2.85$\pm$0.10 -> 2.77$\pm$0.11 | Hurt |
| Increase the number of cloning audio samples | Speaker adaptation and speaker encoding · VCTK | Naturalness and speaker similarity | Improved |
| Use speaker encoding rather than embedding adaptation | Voice cloning · VCTK | Cloning time: $\sim 8$ hours -> $\sim 1.5 - 3.5$ secs | Improved |
| Use speaker encoding instead of embedding adaptation | Voice cloning · VCTK | Speaker verification performance | No clear effect |
| Train speaker encoding jointly from scratch rather than separately predicting trained-model embeddings (ablation) | Speaker encoding · Multi-speaker training speakers | Speaker differences in generated audio | Hurt |
| Add discriminative losses for intermediate embeddings or generated audio (ablation) | Jointly trained speaker encoder · Multi-speaker training speakers | Speaker differences | No clear effect |
| Add attention over cloning audio samples instead of simply averaging them (ablation) | Speaker encoder · Validation set of 25 speakers | Embedding estimation error | Improved |
| Train the multi-speaker model on cleaner VCTK speech rather than LibriSpeech | Speaker adaptation · VCTK | Naturalness | Improved |
| Use embedding-only rather than whole-model adaptation with limited speaker diversity (ablation) | Speaker adaptation · VCTK (84 training speakers) | Naturalness | Hurt |
Closest papers by meaning
- SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICML 2024 · similarity 0.87)
- Phoneme Hallucinator: One-Shot Voice Conversion via Set Expansion (AAAI 2024 · similarity 0.86)
- RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models (NeurIPS 2026 · similarity 0.85)
- Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech (ICML 2025 · similarity 0.84)
- ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis (ICML 2024 · similarity 0.84)
- Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
