Research map / Image generation / Autoregressive and latent generation
Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vevo is a self-supervised zero-shot voice imitation framework that uses VQ-VAE-tokenized HuBERT features (content and content-style tokens) with an AR content-style transformer and a flow-matching acoustic model to control timbre and style independently.
Problem
Voice imitation of timbre and style (accent, emotion) typically relies on annotated data and poorly disentangles timbre from style, making controllable zero-shot generation hard.
Approach
Progressively decouple speech by quantizing HuBERT-Large hidden features with VQ-VAE and using the codebook size as an information bottleneck: K=4096 yields content-style tokens (timbre removed) and K=32 yields content tokens (style largely removed), with duration reduction further removing style; an AR transformer models style and a flow-matching transformer renders timbre.
Key results
- Vevo-Timbre achieves WER 2.968 on AB zero-shot timbre imitation, versus 3.682 for the best baseline FACodec, with S-SIM 0.420 and FPC 0.686.
- Vevo-Voice reaches E-SIM 0.872 and ES-MOS 3.20 on the emotion set, outperforming HierSpeech++, LM-VC, UniAudio, and FACodec on style imitation.
- Vevo-TTS achieves ES-MOS 4.03 on emotion samples, surpassing CosyVoice and MaskGCT on that metric despite training only on 60K hours of audiobooks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the VQ-VAE codebook vocabulary from 32 to 4096 (ablation) | Acoustic model trained on HuBERT VQ-VAE tokens · Zero-shot timbre imitation | WER: 9.731 $\uparrow$ -> 6.908 $\uparrow$ | Helped |
| Use a smaller content-style tokenizer vocabulary, 1024 instead of 4096 (ablation) | Vevo-Voice · Zero-shot voice imitation (conversion) | WER: 15.214 -> 18.523 | Hurt |
| Reduce the content tokenizer vocabulary from 32 to 16 (ablation) | Vevo-Voice · Zero-shot voice imitation (conversion) | WER: 15.214 -> 23.351 | Hurt |
| Remove duration reduction (ablation) | Vevo-Voice · Zero-shot voice imitation (conversion) | WER: 15.214 -> 15.958 | Hurt |
| Remove duration reduction (ablation) | Vevo-Voice · Zero-shot voice imitation (conversion) | DDUR: 0.933 -> 1.698 | Hurt |
| Use Global-guided continuation instead of reference-style-enhanced continuation (ablation) | Vevo-Voice · Zero-shot voice imitation (conversion) | WER: 15.214 -> 16.809 | Hurt |
| Use Global-guided continuation instead of reference-style-enhanced continuation (ablation) | Vevo-Voice · Zero-shot voice imitation (conversion) | inference input length: 100% -> 42% | Helped |
| Use text supervision and ASR content tokens rather than Vevo-Style's self-supervised content tokens (ablation) | Vevo-Style · Accent style imitation (ASR-AC samples) | WER: 3.083 -> 1.550 | Helped |
| Use text supervision and ASR content tokens rather than Vevo-Style's self-supervised content tokens (ablation) | Vevo-Style · Emotion style imitation (Emovox samples) | A-/E-SIM: 0.825 -> 0.800 | Hurt |
| Use Vevo-Voice instead of Vevo-Timbre to imitate reference style as well as timbre | Vevo voice conversion · Zero-shot voice imitation (conversion; ACCENT, EMOTION) | S-SIM to reference: 0.486 -> 0.517 | Helped |
| Use Vevo-Voice instead of Vevo-Timbre to imitate reference style as well as timbre | Vevo voice conversion · Zero-shot voice imitation (conversion; ACCENT, EMOTION) | WER: 12.351 -> 15.214 | Hurt |
| Use Vevo-TTS rather than Voicebox with the same 60K-hour audiobook training data | Zero-shot voice imitation (synthesis) · ACCENT, EMOTION | ES-MOS: 3.61 -> 4.03 | Helped |
Surprising findings
- Vevo-Style, a zero-shot style imitation model trained only on audiobooks, beats accent and emotion conversion baselines trained with parallel or style-labeled data.
- Vevo-Style (ASR) with ASR-derived content tokens improves intelligibility and accent imitation over the HuBERT-VQ-VAE content tokens.
Experimental setup
Datasets: 60K hours English audiobooks Common Voice English internal emotional and accented corpus LibriSpeech Libri-light
Benchmarks: AB CV ACCENT EMOTION LibriSpeech dev-clean (ABX)
Metrics: WER S-SIM A-SIM E-SIM FPC N-MOS SS-MOS AS-MOS ES-MOS PS-MOS
Baselines: HierSpeech++ LM-VC UniAudio FACodec ASR-AC VoiceShop Conv-Speak Emovox VALL-E Voicebox VoiceCraft CosyVoice
Builds on: VQ-VAE HuBERT Voicebox RepCodec BigVGAN
Closest papers by meaning
- StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching (AAAI 2025 · similarity 0.91)
- Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech (ICML 2025 · similarity 0.88)
- Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.88)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
- SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICML 2024 · similarity 0.87)
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.