Research map / Image generation / Autoregressive and latent generation
CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CoVoMix generates zero-shot, multi-talker, multi-round conversational speech by predicting multi-stream HuBERT semantic tokens from dialogue text and rendering a single-channel mixed mel-spectrogram with a flow-matching acoustic model and HiFi-GAN vocoder.
Problem
Zero-shot TTS handles formal reading well but struggles to generate natural multi-speaker dialogue with spontaneous behaviors (laughter, interjections), overlapping speech, turn-taking, and consistent speaker identity across rounds.
Approach
Replace per-utterance synthesis and phoneme alignment with a multi-stream text-to-semantic model (one HuBERT token stream per talker) feeding a flow-matching acoustic model that directly generates a mixed mono mel-spectrogram conditioned on speaker prompts, allowing overlap and interjections in one channel.
Key results
- On monologue, CoVoMix reaches WER 8.95 and MCD 6.04 versus 15.85 and 9.45 for the phoneme baseline, with SIM 0.49 vs 0.42.
- On dialogue, CoVoMix achieves CMOS +0.81 and SMOS +0.60 over CoVoSingle (both significant at p<0.01), but WER rises to 19.84 versus 11.77 for CoVoSingle due to overlapped speech.
- CoVoMix's turn-taking statistics and laughter duration are closer to ground truth than the concatenation-based CoVoSingle.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace phoneme representation with semantic tokens | CoVoSingle vs Baseline · Fisher monologue test | WER: 15.85 -> 9.99 | Helped |
| replace phoneme representation with semantic tokens | CoVoSingle vs Baseline · Fisher monologue test | SIM: 0.42 -> 0.49 | Helped |
| generate a dialogue jointly rather than concatenate utterances | CoVoMix vs CoVoSingle · Fisher dialogue test | WER: 11.77 -> 19.84 | Hurt |
| generate a dialogue jointly rather than concatenate utterances | CoVoMix vs CoVoSingle · Fisher dialogue test | CMOS: 0.00 -> 0.81† | Helped |
| generate a dialogue jointly rather than concatenate utterances | CoVoMix vs CoVoSingle · Fisher dialogue test | SMOS: 0.00 -> 0.60† | Helped |
| add short monologue training data (ablation) | text-to-semantic model · Fisher monologue test | WER: 28.50 -> 11.99 | Helped |
| add short monologue training data (ablation) | text-to-semantic model · Fisher dialogue test | WER: 22.30 -> 21.82 | Helped |
| add simulated dialogue training data (ablation) | text-to-semantic model with real dialogue and short monologue · Fisher monologue test | WER: 11.99 -> 11.22 | Helped |
| add simulated dialogue training data (ablation) | text-to-semantic model with real dialogue and short monologue · Fisher dialogue test | WER: 21.82 -> 20.79 | Helped |
| add long monologue training data (ablation) | text-to-semantic model with real and simulated dialogue and short monologue · Fisher monologue test | WER: 11.22 -> 10.54 | Helped |
| add long monologue training data (ablation) | text-to-semantic model with real and simulated dialogue and short monologue · Fisher dialogue test | WER: 20.79 -> 20.87 | Hurt |
| use stereo acoustic output instead of mixed mono output (ablation) | CoVoMix · Fisher dialogue test | WER: 19.84 -> 20.35 | Hurt |
Surprising findings
- Predicting a single mixed mel-spectrogram yields lower WER-based intelligibility on dialogue than dual-channel output, while dual-channel outputs show higher NISQA.
- The flow-matching acoustic model produced cleaner audio than the ground-truth recordings (NISQA 3.01-3.04 for monologue vs 3.03 ground truth; dialogue 2.87-2.90 vs 2.73).
Limitations
- Words are occasionally omitted or duplicated because the autoregressive text-to-semantic model lacks forced duration.
- Fisher data is 8 kHz telephone speech with background noise, degrading quality.
- Overlapped speech inflates WER, making intelligibility evaluation on dialogue unreliable.
Experimental setup
Datasets: Fisher
Benchmarks: Fisher test split
Metrics: SIM WER MCD NISQA CMOS SMOS turn-taking statistics laughter count and duration speech consistency
Baselines: Baseline phoneme flow-matching TTS (VoiceBox-like) CoVoSingle dGSLM CHATS SoundStorm
Builds on: VoiceBox dGSLM HuBERT HiFi-GAN conditional flow matching
Closest papers by meaning
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching (NeurIPS 2025 · similarity 0.94)
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction (ICML 2025 · similarity 0.88)
- TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (NeurIPS 2025 · similarity 0.87)
- MoonCast: High-Quality Zero-Shot Podcast Generation (NeurIPS 2025 · similarity 0.87)
- Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.87)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.