Research map / Image generation / Editing and style transfer
StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StableVC, a non-autoregressive, style-controllable zero-shot voice conversion model that disentangles content, timbre and style and uses a conditional flow matching DiT decoder with a gated dual attention (DualAGC) to convert timbre and style from different unseen speakers.
Problem
Zero-shot voice conversion typically transfers only timbre, with style tied to the prompt; existing systems are slow (autoregressive or many-step diffusion) and lack independent timbre and style control from different unseen speakers.
Approach
Decomposes speech into WavLM K-means content, factorized-codec style and mel-spectrogram timbre, and reconstructs mel with a conditional flow matching DiT in which a dual attention block (timbre attention with speaker-embedding prior plus style attention gated by a zero-initialized tanh(alpha)) replaces feature concatenation; GRL removes timbre leakage from style.
Key results
- StableVC reaches UTMOS 4.12, WER 2.03, SECS 0.67 and nMOS 3.96, outperforming all six zero-shot VC baselines in the main table (best baseline SEF-VC: UTMOS 3.59, WER 2.89, SECS 0.53).
- StableVC achieves about 25.3x speedup over the language-model-based VALLE-VC and about 1.65x over the diffusion-based DDDM-VC in real-time factor (0.146 with 10 Euler steps).
- In zero-shot style transfer with timbre and style from different unseen speakers, StableVC gets Corr 0.75 and RMSE 12.87 versus 0.71/14.98 for the best baseline StyleVC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use StableVC instead of the strongest reported baseline | Zero-shot VC · VCTK zero-shot VC | UTMOS: 3.75 -> 4.12 | Helped |
| Use StableVC instead of the strongest reported baseline | Zero-shot VC · VCTK zero-shot VC | SECS: 0.61 -> 0.67 | Helped |
| Use StableVC instead of the strongest reported baseline | Zero-shot VC · VCTK zero-shot VC | RTF: 0.075 -> 0.146 | Hurt |
| Use StableVC instead of the strongest reported baseline | Zero-shot style transfer · Zero-shot style transfer | Corr: 0.71 -> 0.75 | Helped |
| Use StableVC instead of the strongest reported baseline | Zero-shot style transfer · Zero-shot style transfer | WER: 5.31 -> 2.12 | Helped |
| Use 10 rather than 1 Euler step (ablation) | StableVC · VCTK zero-shot VC | UTMOS: 3.42 -> 4.12 | Helped |
| Use 10 rather than 1 Euler step (ablation) | StableVC · VCTK zero-shot VC | RTF: 0.031 -> 0.146 | Hurt |
| Add timbre prior and multiple-reference speech (ablation) | StableVC · Zero-shot style transfer | WER: 22.16 -> 2.12 | Helped |
| Add timbre prior and multiple-reference speech (ablation) | StableVC · Zero-shot style transfer | SECS: 0.41 -> 0.64 | Helped |
| Add adaptive gate α (ablation) | StableVC · Zero-shot style transfer | SECS: 0.36 -> 0.64 | Helped |
| Add GRL loss (ablation) | StableVC · Zero-shot style transfer | UTMOS: 3.62 -> 4.06 | Helped |
| Add deduplication and duration modeling (ablation) | StableVC · Zero-shot style transfer | Corr: 0.66 -> 0.75 | Helped |
Surprising findings
- Removing the timbre prior and multi-reference input raised WER from 2.12 to 22.16, showing the timbre attention is highly sensitive to these inputs.
- StyleVC has the lowest RTF (0.075) and fewest parameters but performs poorly on most quality and similarity metrics.
Limitations
- Style transfer evaluation uses F0 from the style reference as the style prompt for baselines lacking style support, which is a constructed comparison.
- Dependence on several pretrained external components (WavLM, factorized codec, speaker verification model, vocoder) whose quality bounds performance.
- Evaluation relies on a small set of 20 timbre and 200 style references and subjective MOS scores, limiting statistical breadth.
Experimental setup
Datasets: LibriLight VCTK ESD
Benchmarks: VCTK ESD
Metrics: nMOS sMOS-p sMOS-s UTMOS WER SECS RTF Corr RMSE
Baselines: StyleVC LM-VC VALLE-VC NS2VC DDDM-VC SEF-VC
Builds on: flow matching DiT VoiceFlow Matcha-TTS Voicebox
Closest papers by meaning
- Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.91)
- Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.89)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.87)
- Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech (ICML 2025 · similarity 0.87)
- Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion (AAAI 2025 · similarity 0.87)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.