Research map / Label-efficient and robust learning / Self-supervised and contrastive
SelfVC: Voice Conversion With Iterative Refinement using Self Transformations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelfVC, which iteratively refines a text-free, SSL-based voice conversion synthesizer by training it on self-synthesized voice-converted inputs in addition to heuristic perturbations.
Problem
Zero-shot voice conversion from imperfectly disentangled SSL speech representations, where heuristic perturbations give limited diversity of speaker-varying input transformations and prosody control is hard without text.
Approach
Use the synthesizer's own voice-converted outputs (with a random other speaker's embedding) as challenging perturbed content inputs for training, iteratively replacing heuristic transforms after a warm-up, with a duration-augmented content embedding from grouped SSL features for prosody control.
Key results
- On zero-shot LibriTTS VC, SelfVC reaches SV-EER 3.4% vs 6.0% for Baseline-Heuristic and 6.6% for ACE-VC and YourTTS, with Sim-MOS 3.74 and MOS 4.06.
- In cross-lingual English-to-CSS10 VC, SelfVC (LibriTTS + CSS10) reduces SV-EER to 4.4% vs 13.2% for YourTTS.
- On VCTK unseen speakers, SelfVC achieves SV-EER 4.3% vs 8.5% for Baseline-Heuristic and 9.2% for ACE-VC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add heuristic input transformations (ablation) | Baseline–NoTransform voice conversion synthesizer · LibriTTS test-clean zero-shot voice conversion | SV-EER: 28.9% -> 6.0% | Helped |
| add self-transformation iterative refinement after heuristic training (ablation) | Baseline–Heuristic voice conversion synthesizer · LibriTTS test-clean zero-shot voice conversion | SV-EER: 6.0% -> 3.4% | Helped |
| add self-transformation iterative refinement after heuristic training (ablation) | Baseline–Heuristic voice conversion synthesizer · LibriTTS test-clean zero-shot voice conversion | SV-Sim: 0.53 -> 0.58 | Helped |
| add self-transformation iterative refinement after heuristic training (ablation) | Baseline–Heuristic voice conversion synthesizer · LibriTTS test-clean zero-shot voice conversion | Sim-MOS: 3.65 ± 0.07 -> 3.74 ± 0.07 | Helped |
| fine-tune the synthesis model on CSS10 multilingual data | SelfVC (LibriTTS) · CSS10 to CSS10 | SV-EER: 18.4% -> 4.4% | Helped |
| fine-tune the synthesis model on CSS10 multilingual data | SelfVC (LibriTTS) · CSS10 to CSS10 | PER: 23.7% -> 18.9% | Helped |
| fine-tune the synthesis model on CSS10 multilingual data | SelfVC (LibriTTS) · CSS10 to English | SV-EER: 7.0% -> 4.8% | Helped |
| fine-tune the synthesis model on CSS10 multilingual data | SelfVC (LibriTTS) · CSS10 to English | PER: 25.4% -> 19.9% | Hurt |
| use SelfVC trained on English LibriTTS rather than multilingual YourTTS | zero-shot voice conversion · CSS10 to CSS10 | SV-EER: 12.5% -> 18.4% | Hurt |
| use SelfVC trained on English LibriTTS rather than multilingual YourTTS | zero-shot voice conversion · CSS10 to CSS10 | PER: 23.6% -> 23.7% | Mixed |
| use SelfVC instead of YourTTS | zero-shot voice conversion · English to CSS10 | SV-EER: 13.2% -> 12.7% | Helped |
| use SelfVC instead of YourTTS | zero-shot voice conversion · CSS10 to English | SV-EER: 9.7% -> 7.0% | Helped |
Surprising findings
- Baseline-NoTransform yields intelligible, natural audio but very poor speaker similarity (SV-EER 28.9%), showing transformations are essential for voice conversion.
- SelfVC (LibriTTS) achieves better speaker similarity than YourTTS despite being trained only on LibriTTS English, while YourTTS uses a much larger multilingual dataset.
Limitations
- Speaker similarity lags on CSS10-to-CSS10 conversion compared with YourTTS, which was trained on multilingual data.
- Cross-lingual gains rely on fine-tuning on CSS10 speakers, which the paper notes makes the SV-EER improvement less indicative of unseen-speaker generalization.
- Training is English-focused for the primary model; PER is higher for languages with large phonetic inventories such as Chinese, Russian and Japanese.
Experimental setup
Datasets: LibriTTS train-clean-360 LibriLight CSS10 VCTK LibriTTS test-clean
Benchmarks: LibriTTS test-clean voice conversion trials VCTK CSS10 cross-lingual LibriTTS train-clean-360 seen speakers
Metrics: SV-EER SV-Sim Sim-MOS PER CER MOS GPE
Baselines: AdaIN-VC MediumVC FragmentVC S3PRL-VC YourTTS ACE-VC Baseline-NoTransform Baseline-Heuristic
Builds on: NANSY FastPitch HiFi-GAN TitaNet ContentVec
Closest papers by meaning
- Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.87)
- Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion (AAAI 2025 · similarity 0.87)
- Phoneme Hallucinator: One-Shot Voice Conversion via Set Expansion (AAAI 2024 · similarity 0.86)
- Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction (AAAI 2024 · similarity 0.86)
- DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion (AAAI 2024 · similarity 0.86)
- Improving Generalization for AI-Synthesized Voice Detection (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.