Research map / Image generation / Autoregressive and latent generation
MoonCast: High-Quality Zero-Shot Podcast Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoonCast, a zero-shot two-speaker podcast generator that pairs an LLM-written spontaneous script with a 2.5B long-context language-model audio generator and a chunk-wise autoregressive flow-matching detokenizer.
Problem
Zero-shot TTS systems struggle to produce long (several-minute), multi-speaker, spontaneous podcast-style speech from text-only sources, due to long speech sequences and the lack of conversational spontaneity.
Approach
Model the whole multi-turn two-speaker podcast as one long sequence with a 2.5B Llama-style LM over semantic codes (with speaker-change tokens and per-speaker prompts), trained with three-stage curriculum learning, plus an LLM script generator that injects filler words and other spontaneous details, and a chunk-wise causal flow-matching detokenizer for long-form inference.
Key results
- On Chinese podcasts, MoonCast reaches spontaneity 4.25 and coherence 4.13 vs. 3.85 and 3.80 for the concatenation baseline and 3.63 and 3.38 for CosyVoice2 (5-point subjective scores).
- On English podcasts, MoonCast reaches spontaneity 4.63 and coherence 4.50 vs. 3.78 and 3.80 for CosyVoice2, with CER 1.91 (best).
- Spontaneous LLM-written scripts outperform written scripts on spontaneity (4.03 vs. 3.21) and quality (4.04 vs. 3.61) in the script-influence study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · Chinese podcast generation | Spontaneity: 3.85 -> 4.25 | Helped |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · Chinese podcast generation | Coherence: 3.80 -> 4.13 | Helped |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · Chinese podcast generation | Intelligibility: 4.33 -> 4.38 | Helped |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · Chinese podcast generation | Quality: 3.68 -> 3.80 | Helped |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · Chinese podcast generation | SIM-O: 0.86 -> 0.77 | Hurt |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · Chinese podcast generation | CER: 1.90 -> 2.15 | Hurt |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · English podcast generation | Spontaneity: 3.58 -> 4.63 | Helped |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · English podcast generation | Coherence: 3.58 -> 4.50 | Helped |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · English podcast generation | Intelligibility: 3.93 -> 4.63 | Helped |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · English podcast generation | SIM-O: 0.75 -> 0.53 | Hurt |
| Use long-context two-speaker audio modeling instead of turn-by-turn concatenation | MoonCast · English podcast generation | CER: 2.57 -> 1.91 | Helped |
| Remove spontaneous details from the script (ablation) | MoonCast with GT script · Chinese podcast generation | Spontaneity: 4.16 -> 3.21 | Hurt |
Surprising findings
- MoonCast has lower SIM-O than baselines on both Chinese (0.77 vs 0.86 for concatenation) and English (0.53 vs 0.75), with the authors attributing this to missing audiobook data for English.
- Written scripts cause a training-inference mismatch that degrades quality, similarity, and coherence, showing script spontaneity matters as much as the TTS model.
Limitations
- Hallucinations in speaker identity attribution, partly caused by semantic tokens retaining timbre and ambiguous text such as filler words.
- Trade-off between sampling diversity for spontaneity and increased hallucinations.
- Lower SIM-O for English podcasts due to lack of audiobook data for long-context English speech.
Experimental setup
Datasets: internal Chinese and English audio dataset (podcasts, audiobooks) Fisher dataset (referenced in related work only)
Benchmarks: internal Chinese podcast evaluation set (7 podcasts, 125 turns) internal English podcast evaluation set (PDF and web URL sources)
Metrics: spontaneity (MOS) coherence (MOS) intelligibility (MOS) quality (MOS) similarity (MOS) SIM-O CER
Baselines: CosyVoice2 Concatenation baseline
Builds on: MaskGCT CosyVoice VALL-E ARDit BigVGAN
Closest papers by meaning
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching (NeurIPS 2025 · similarity 0.88)
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · similarity 0.87)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.84)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.84)
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.84)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.