Research map / Image generation / Autoregressive and latent generation
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniCATS, a unified context-aware TTS framework using contextual VQ-diffusion (CTX-txt2vec) to predict vq-wav2vec semantic tokens and a contextual vocoder (CTX-vec2wav) conditioned on mel-spectrogram prompts, handling both speech continuation and editing.
Problem
Existing discrete-token TTS models such as VALL-E and SPEAR-TTS are autoregressive, so they cannot condition on both preceding and following context for speech editing, and their audio quality is bounded by acoustic codec tokens.
Approach
Replace AR acoustic-token generation with contextual VQ-diffusion over semantic tokens that concatenates clean left/right context tokens with corrupted target tokens, and use a vocoder that takes speaker identity from a mel-spectrogram prompt via cross-attention instead of x-vectors or acoustic tokens.
Key results
- CTX-vec2wav reaches naturalness MOS 4.75 and SECS 0.845 for semantic-token resynthesis, above HifiGAN (4.30, 0.776) and AudioLM (3.99, 0.801).
- UniCATS gets unseen-speaker naturalness MOS 4.43 and similarity MOS 4.25 in zero-shot continuation, above VALL-E (4.17, 3.83).
- In speech editing, UniCATS scores MOS 4.62 (short) and 4.63 (long), above RetrieverTTS (4.43 and 4.37).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use contextual vocoding instead of HifiGAN semantic-token vocoding | Semantic-token speech resynthesis · LibriTTS test set B | Naturalness MOS: 4.30 $\pm$ 0.08 -> 4.75 $\pm$ 0.06 | Helped |
| Use contextual vocoding instead of HifiGAN semantic-token vocoding | Semantic-token speech resynthesis · LibriTTS test set B | SECS: 0.776 -> 0.845 | Helped |
| Use UniCATS instead of VALL-E | Zero-shot speaker-adaptive TTS · LibriTTS seen speakers | Naturalness MOS: 4.23 $\pm$ 0.07 -> 4.54 $\pm$ 0.07 | Helped |
| Use UniCATS instead of VALL-E | Zero-shot speaker-adaptive TTS · LibriTTS seen speakers | Similarity MOS: 3.98 $\pm$ 0.06 -> 4.56 $\pm$ 0.07 | Helped |
| Use UniCATS instead of VALL-E | Zero-shot speaker-adaptive TTS · LibriTTS unseen speakers | Naturalness MOS: 4.17 $\pm$ 0.09 -> 4.43 $\pm$ 0.08 | Helped |
| Use UniCATS instead of VALL-E | Zero-shot speaker-adaptive TTS · LibriTTS unseen speakers | Similarity MOS: 3.83 $\pm$ 0.07 -> 4.25 $\pm$ 0.08 | Helped |
| Use UniCATS instead of RetrieverTTS | Speech editing · LibriTTS test set C, short segments | MOS: 4.43 $\pm$ 0.08 -> 4.62 $\pm$ 0.06 | Helped |
| Use UniCATS instead of RetrieverTTS | Speech editing · LibriTTS test set C, long segments | MOS: 4.37 $\pm$ 0.08 -> 4.63 $\pm$ 0.06 | Helped |
| Increase editing segment length | RetrieverTTS · LibriTTS test set C | MOS: 4.43 $\pm$ 0.08 -> 4.37 $\pm$ 0.08 | Hurt |
| Increase editing segment length | UniCATS · LibriTTS test set C | MOS: 4.62 $\pm$ 0.06 -> 4.63 $\pm$ 0.06 | No clear effect |
| Replace vq-wav2vec tokens with HuBERT tokens (ablation) | CTX-vec2wav · LibriTTS test set B resynthesis | WER: 3.64 -> 2.73 | Helped |
| Use vq-wav2vec tokens instead of HuBERT tokens (ablation) | CTX-vec2wav · LibriTTS test set B resynthesis | WER: 2.73 -> 3.64 | Hurt |
Surprising findings
- Vq-wav2vec tokens, chosen for richer prosody, yield higher WER (3.64) than HuBERT tokens (2.73) in resynthesis, so neither token type dominates.
Limitations
- Speaker similarity relies on a short mel prompt, and the paper notes longer prompts may help but were not explored.
- Vq-wav2vec tokens give higher WER than HuBERT tokens, so semantic token choice trades off intelligibility against prosody.
- VALL-E and AudioLM were evaluated via open-source reimplementations rather than official models.
Experimental setup
Datasets: LibriTTS
Benchmarks: LibriTTS test set A LibriTTS test-clean test set B RetrieverTTS editing test set C
Metrics: MOS naturalness MOS similarity SECS WER
Baselines: VALL-E AudioLM HifiGAN FastSpeech 2 RetrieverTTS Encodec
Builds on: VQ-Diffusion HifiGAN FastSpeech 2 vq-wav2vec Conformer
Closest papers by meaning
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.87)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.87)
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.86)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.86)
- SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models (ICLR 2024 · similarity 0.86)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.