Research map / Image generation / Autoregressive and latent generation

MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer

ICLR 2025 · Image generation › Autoregressive and latent generation · approach: Neural text-to-speech synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MaskGCT, a fully non-autoregressive two-stage zero-shot TTS system using masked generative transformers on VQ-VAE semantic tokens (from W2v-BERT 2.0) and RVQ acoustic tokens, trained on 100K hours of Emilia without alignment or phone-level duration.

Problem

Zero-shot TTS systems are either autoregressive (poor robustness, slow, no duration control) or non-autoregressive (require text-speech alignment and phone-level duration prediction, yielding standardized speech).

Approach

Use mask-and-predict masked generative transformers for both text-to-semantic and semantic-to-acoustic stages, with in-context prompting via prefixes, a VQ-VAE semantic codec instead of k-means, and a flow-matching total-duration predictor, eliminating alignment supervision and phone-level duration prediction.

Key results

• On LibriSpeech test-clean MaskGCT reaches SIM-O 0.687 (0.697 with gt length) vs 0.67 for NaturalSpeech 3, with CMOS +0.10 vs ground truth.
• On SeedTTS test-en, MaskGCT with gt length reaches SIM-O 0.728 and WER 2.466; on test-zh SIM-O 0.777 and WER 2.183.
• Compared with AR+SoundStorm, MaskGCT improves CMOS by +0.12, +0.08, and +0.37 across the three main test sets.

Measured effects reported in the paper

Surprising findings

• Fewer S2A inference steps ([10,1,...]) lose only slightly in performance, and the T2S model needs only about 25 steps for best WER.
• BPE tokenization gives lower WER than G2P on Chinese (1.921 vs 2.183) even though G2P has higher SIM-O there.

Limitations

• Editing and voice conversion are not robust; editing relies on a text-speech aligner and was only shown in demos.
• Cross-lingual translation has high WER for some target languages (Zh, De, Fr) and multilingual results retrain only the T2S model.
• Duration predictor is a separate flow-matching model; total duration is estimated rather than fully implicit.

Experimental setup

Datasets: Emilia LibriSpeech Seed-TTS test sets ESD L2-ARCTIC

Benchmarks: LibriSpeech test-clean SeedTTS test-en SeedTTS test-zh SeedTTS test-hard L2-ARCTIC ESD

Metrics: SIM-O WER FSD SMOS CMOS SIM-Accent Emotion SIM

Baselines: VALL-E VoiceBox NaturalSpeech 3 VoiceCraft XTTS-v2 CosyVoice AR + SoundStorm

Builds on: SoundStorm VQ-VAE DAC Vocos NaturalSpeech 3

Closest papers by meaning

• CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.89)
• Masked Audio Generation using a Single Non-Autoregressive Transformer (ICLR 2024 · similarity 0.88)
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-Guided Sequence Reordering (AAAI 2025 · similarity 0.87)
• DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.87)
• Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.86)