Research map / Image generation / Autoregressive and latent generation
CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoVoMix2, a fully non-autoregressive flow-matching model that generates multi-speaker dialogue mel-spectrograms directly from disentangled multi-stream character transcriptions with zero-shot speaker prompts.
Problem
Zero-shot multi-talker dialogue generation struggles with speaker consistency, overlapping speech, controllable timing, and inference speed in existing AR or hybrid AR+NAR systems that rely on intermediate tokens and transcribed prompts.
Approach
Directly predict mel-spectrograms from parallel per-speaker character streams (with silence and continuation tokens) using a flow-matching transformer, with sentence-level character-frame alignment, prompt-level random masking of speaker prompts (no prompt transcription needed), and loss masking on the prompt region.
Key results
- CoVoMix2 achieves RTF 0.30 versus 1.37 for MoonCast and 2.08 for Sesame on dialogue data.
- CoVoMix2 attains the best SA-WER (6.31) and SA-SIM (0.56) and UTMOS (3.10) on the dialogue test set, versus Sesame's SA-WER 9.65 and MoonCast's 20.40.
- In real-world prompt test, CoVoMix2 reaches SA-WER 9.32 versus 26.26 for MoonCast and 25.19 for Sesame.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MoonCast with CoVoMix2 | Dialogue generation · Dialogue test set | RTF: 1.37 -> 0.30 | Helped |
| Replace Sesame with CoVoMix2 | Dialogue generation · Dialogue test set | SA-WER: 9.65$\pm$13.20 -> 6.31$\pm$9.24 | Helped |
| Replace Sesame with CoVoMix2 | Dialogue generation · Dialogue test set | WER: 5.62$\pm$5.61 -> 5.73$\pm$6.68 | No clear effect |
| Replace MoonCast or Sesame with CoVoMix2 | Monologue generation · Audiobook-style monologue test set | UTMOS: 2.82$\pm$0.48 -> 3.19$\pm$0.37; 2.81$\pm$0.52 -> 3.19$\pm$0.37 | Helped |
| Remove monologue pretraining (ablation) | CoVoMix2 · Two-speaker test set | WER: 5.67 -> 82.66 | Hurt |
| Add 2,000-step fine-tuning on 20 minutes of human-annotated dialogue (ablation) | CoVoMix2 with pretraining · Two-speaker test set | SA-WER: 6.21 -> 3.81 | Helped |
| Add monologue podcast data to dialogue podcast data (ablation) | CoVoMix2 training data · Dialogue test set | SA-WER: 8.63 -> 8.51 | Helped |
| Add simulated dialogue data (ablation) | CoVoMix2 training data · Dialogue test set | SA-WER: 8.51 -> 9.31 | Hurt |
| Add LibriHeavy audiobook data (ablation) | CoVoMix2 training data · Dialogue test set | SA-WER: 9.31 -> 7.43 | Helped |
| Add LibriTTS audiobook data (ablation) | CoVoMix2 training data · Dialogue test set | SA-WER: 7.43 -> 6.58 | Helped |
| Use [S] instead of [P] for silence (ablation) | CoVoMix2 · Dialogue test set | SA-WER: 7.09 -> 5.59 | Helped |
| Use speaker-aware [S1,S2] instead of generic [S] (ablation) | CoVoMix2 · Dialogue test set | SA-WER: 5.59 -> 6.58 | Hurt |
Surprising findings
- Simulated overlapping dialogue data does not consistently improve performance and may slightly degrade it, though it is needed for overlap capability.
- Training without the monologue pretraining stage yields WER above 80%, showing dialogue-only training is unstable.
Limitations
- Training data is primarily ASR-transcribed and may contain transcription errors, especially disfluencies and backchannels.
- Overlap training relies on simulated dialogues that can introduce naturalness and audio quality deviations.
- Experiments are limited to two-speaker dialogues and English.
Experimental setup
Datasets: LibriHeavy LibriTTS DailyDialog LibriSpeech test-clean Deepgram ASR transcriptions of internal podcasts
Benchmarks: CoVoMix2 dialogue test set (DailyDialog text) audiobook-style monologue test set LibriSpeech-based prompt length test NCSSD-CEN real-world prompt test
Metrics: RTF WER SA-WER SA-SIM UTMOS CMOS SIM
Baselines: MoonCast Sesame
Builds on: CoVoMix E2-TTS F5-TTS Classifier-Free Guidance
Closest papers by meaning
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · similarity 0.94)
- MoonCast: High-Quality Zero-Shot Podcast Generation (NeurIPS 2025 · similarity 0.88)
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction (ICML 2025 · similarity 0.88)
- Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.87)
- TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (NeurIPS 2025 · similarity 0.86)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.