Research map / Probabilistic and scientific modeling / Variational and generative sampling
Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GMM-LM, a continuous autoregressive TTS model using a Gaussian-mixture VAE latent space (no quantization) and a stochastic hard monotonic alignment, achieving lower WER and higher MOS than VALL-E with about 10% of its parameters.
Problem
Autoregressive speech synthesis relies on discrete codecs (VQ/RVQ), which suffer from large codebook requirements, mispronunciations, under-utilized codes, and multi-stage modeling overhead; AR TTS models like VALL-E also suffer from word skipping and repetition.
Approach
Replace quantization with a VAE whose continuous latent prior is a Gaussian mixture, and model the continuous latents autoregressively with a GMM output head; add a stochastic hard monotonic attention using Bernoulli sampling in the forward pass and a Gumbel-Softmax relaxation in the backward pass.
Key results
- GMM-LM Large reaches 2.81% WER and 0.85 SIM at 3-second prompts vs VALL-E's 6.04% WER and 0.73 SIM.
- GMM-LM Mini (51.5M parameters, about 10.3% of VALL-E) achieves 2.83% WER and Q-MOS 4.11 vs VALL-E's 3.54 at 3-second prompts.
- Monotonic alignment with ST Gumbel lowers WER to 2.72% compared with 6.6% for plain cross-attention on the GMM-LM ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GMM-LM with continuous GMM-VAE features instead of Mel-spectrogram features | GMM-LM · LibriSpeech test, zero-shot TTS | WER and SIM | Helped |
| Scale GMM-LM from Mini to Large | GMM-LM · LibriSpeech test, zero-shot TTS | SIM and S-MOS: 0.80 -> 0.85 SIM; 3.84 -> 4.04 S-MOS | Helped |
| Scale GMM-LM from Mini to Large | GMM-LM · LibriSpeech test, zero-shot TTS | WER: 2.83 -> 2.81 | No clear effect |
| Use ST Gumbel monotonic alignment instead of cross-attention (ablation) | GMM-LM · LibriSpeech test, alignment ablation | WER (%): 6.6 -> 2.72 | Helped |
| Use ST Gumbel monotonic alignment instead of location-sensitive attention (ablation) | GMM-LM · LibriSpeech test, alignment ablation | WER (%): 6.02 -> 2.72 | Helped |
| Use ST Gumbel instead of pre-sigmoid noise for monotonic attention (ablation) | GMM-LM · LibriSpeech test, alignment ablation | WER (%): 3.77 -> 2.72 | Helped |
| Use ST Gumbel instead of Gumbel for monotonic attention (ablation) | GMM-LM · LibriSpeech test, alignment ablation | WER (%): 3.34 -> 2.72 | Helped |
| Add the proposed monotonic alignment to a discrete AR model (ablation) | Discrete AR · LibriSpeech-test-clean | WER (%): 8.02 -> 5.35 | Helped |
| Add the proposed monotonic alignment to discrete AR with delayed codebook prediction (ablation) | Discrete AR with Delay Pred. · LibriSpeech-test-clean | WER (%): 7.83 -> 5.87 | Helped |
| Increase divergence weight λ from 1 to 50 (ablation) | GMM-LM with 3-Gaussian GMM-VAE features · LibriSpeech test, teacher-forced reconstruction | Mel Distance: 3.04 -> 1.82 | Helped |
| Increase divergence weight λ from 1 to 50 (ablation) | GMM-VAE with 3 Gaussians · LibriSpeech test, codec reconstruction | Mel Distance: 0.77 -> 1.13 | Hurt |
| Use three Gaussians instead of one in the divergence constraint (ablation) | GMM-VAE and GMM-LM · LibriSpeech test, reconstruction | Mel Distance: GMM-VAE λ=50: 1.67 -> 1.13; GMM-LM λ=50: 3.16 -> 1.82 | Helped |
Surprising findings
- Without any divergence constraint on the GMM-VAE latent, downstream GMM-LM models could not be trained successfully.
- VALL-E's WER worsened as prompt length increased (6.04% at 3s to 9.68% at 15s).
Limitations
- Only the GMM means are learned; mixture weights are uniform and covariances fixed to identity in the GMM-VAE prior.
- Large model gains mainly appear in speaker metrics; WER and quality improve little with scale.
- Larger 6-mixture GMM-VAE appears to overfit, and the 1-mixture and 10-mixture settings underperform.
Experimental setup
Datasets: LibriLight LibriSpeech test-clean VoxCeleb
Benchmarks: LibriSpeech test-clean
Metrics: WER SIM S-MOS Q-MOS Mel Distance ViSQOL SI-SDR Diversity
Baselines: VALL-E HierSpeech++ StyleTTS2 Encodec Discrete AR (VQ-VAE) Discrete AR with delay prediction
Builds on: VALL-E VQ-VAE DAC VITS Encodec
Closest papers by meaning
- DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.89)
- Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space (NeurIPS 2025 · similarity 0.88)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.87)
- DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.87)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.86)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.