Research map / Label-efficient and robust learning / Recognition and long-tail

A Variational Framework for Improving Naturalness in Generative Spoken Language Models

ICML 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Multilingual language representation pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a VAE with an autoregressive prior that learns continuous variational features jointly with HuBERT semantic tokens to improve naturalness of generative spoken language models, without hand-engineered pitch features.

Problem

Token-based generative spoken language models using semantic tokens neglect prosody and paralinguistic information, producing less natural speech; prior fixes rely on hand-engineered features like pitch.

Approach

Jointly train a VAE whose prior is an autoregressive model (with a time-wise normalizing flow) over learned continuous variational features that complement discrete semantic tokens, optimizing reconstruction plus autoregressive prediction losses, with a diffusion decoder conditioned on both.

Key results

• Proposed approach achieves N-MOS 3.60 vs 3.19 for Token-LM and M-MOS 3.45 vs 3.24 for Token-LM on Libri-light speech continuation.
• Reconstruction F0-RMSE 16.56 and CER 4.35 for proposed vs 43.90 and 10.19 for Token-LM; Token-LM + Acoustic achieves best reconstruction (CER 3.73).
• Proposed model adds about 221M parameters versus 219M for Token-LM, with sWUGGY 60.48 vs 61.75 for Token-LM (slightly lower).

Measured effects reported in the paper

Surprising findings

• Removing semantic tokens slightly improves sWUGGY (69.33 vs 66.46 with tokens on LibriSpeech) but markedly worsens CER (13.02), suggesting sWUGGY may exploit prosodic cues.
• Token-LM + Acoustic tokens yields the best reconstruction but the lowest speech continuation naturalness and meaningfulness among the methods.

Limitations

• Performance is sensitive to the loss-balancing hyperparameters beta and gamma.
• Evaluation limited to English audiobook data; generalization to other languages and prosodic patterns is untested.
• Small model and training data compared to existing speech language model frameworks.

Experimental setup

Datasets: LibriSpeech Libri-light EmoV-DB VCTK

Benchmarks: LibriSpeech dev-clean LibriSpeech dev-other sWUGGY sBLIMP EmoV-DB VCTK

Metrics: F0-RMSE MCD CER sWUGGY sBLIMP M-MOS N-MOS emotion recognition accuracy speaker identification accuracy

Baselines: Token-LM Token-LM + Pitch Token-LM + Acoustic SpeechTokenizer-LM

Builds on: GSLM beta-VAE HuBERT DDPM DDIM

Closest papers by meaning

• Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.88)
• GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling (ICLR 2025 · similarity 0.87)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
• Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.86)
• Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.86)
• Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.86)