Research map / Probabilistic and scientific modeling / Molecules and proteins
Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLED, a purely autoregressive speech language model over continuous Encodec-derived latents that uses an energy distance loss with a lightweight MLP generative head, instead of RVQ tokens or iterative per-step diffusion.
Problem
Autoregressive speech language modeling is hampered by discretization errors and the hierarchical architectures needed for residual VQ token streams; continuous alternatives need a per-step generative module that is expressive, stable, and fast to sample.
Approach
Model each continuous latent step with a lightweight implicit generator (MLP with AdaLN noise injection) trained by a generalized energy distance (beta=1 L2 distance) that combines an RMSE-like term with a repulsive term, making the loss a strictly proper scoring rule while requiring only one forward pass per sample.
Key results
- Zero-shot TTS with 3s prefix prompt: SLED reaches WER-C 1.59 and WER-H 1.99 with 0.2B params, versus VALL-E WER-H 3.8 and ground truth WER-C 1.78.
- Reference-utterance prompting: SLED achieves WER-C 1.51, SIM 0.664, outperforming VALL-E (WER-H 5.9) and matching MELLE's SIM.
- Replacing the energy distance objective with RMSE raises WER-C from 1.59 to 40.60, showing the repulsive term is essential.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Energy Distance with Root Mean Squared Error (ablation) | SLED · LibriSpeech-PC test-clean, 3s Prefix as Prompt | WER-C (%): 1.59 -> 40.60 | Hurt |
| Add classifier-free guidance at λ=1.5 instead of no guidance (λ=1.0) | SLED streaming inference · Streaming inference | WER-C (%): 6.01 -> approximately 2.0 | Helped |
| Increase classifier-free guidance strength beyond λ=1.5 | SLED · Generation settings | Timbre cloning and perceived speech quality | Hurt |
| Use classifier-free guidance rather than no guidance | SLED · Generation settings | Text alignment / WER-C: 6.01 -> approximately 2.0 | Helped |
| Use streaming inference instead of offline inference | SLED · Speech synthesis | WER-C (%): 1.67 -> 2.18 (5:20); 1.67 -> 2.20 (5:45) | Hurt |
| Use streaming inference instead of offline inference | SLED · Speech synthesis | DNSMOS: 3.58 -> 3.59 (5:20); 3.58 -> 3.54 (5:45) | No clear effect |
| Change the streaming interleaving ratio from 5:20 to 5:45 | SLED streaming inference · Streaming inference | WER-C (%): 2.18 -> 2.20 | No clear effect |
| Change the streaming interleaving ratio from 5:20 to 5:45 | SLED streaming inference · Streaming inference | DNSMOS: 3.59 -> 3.54 | No clear effect |
| Increase training data from 1,000-hour LibriSpeech to 50,000-hour LibriHeavy | SLED · 3s Prefix as Prompt | WER-C (%): 1.68 -> 1.59 | Helped |
| Increase training data from 1,000-hour LibriSpeech to 50,000-hour LibriHeavy | SLED · Reference Utterance as Prompt | WER-C (%): 2.27 -> 1.51 | Helped |
| Increase inference batch size from 1 to 16 | SLED · Inference | RTF: 0.8 -> 0.055 | Helped |
| Increase inference batch size from 1 to 64 | SLED · Inference | RTF: 0.8 -> 0.027 | Helped |
Surprising findings
- Plain RMSE regression loss causes the model to largely fail (WER-C 40.60), with the model failing to generate speech in most cases.
- SLED's per-step output distribution fails the Shapiro-Wilk normality test in all 128 latent dimensions, arguing against Gaussian per-step heads.
Limitations
- Current latent encoder is the Encodec encoder designed for codec use rather than a dedicated continuous latent encoder.
- Still a significant gap in voice cloning versus traditional TTS models at similar scale.
- Evaluated only on speech synthesis; general-purpose speech language modeling is future work.
Experimental setup
Datasets: LibriHeavy LibriSpeech LibriSpeech-PC
Benchmarks: LibriSpeech test-clean (LibriSpeech-PC)
Metrics: WER-C WER-H SIM DNSMOS RTF FLOPs
Baselines: VALL-E VALL-E 2 ELLA-V VALL-E R Llasa MELLE FELLE CLaM-TTS MaskGCT F5-TTS MegaTTS 3 DiTAR
Builds on: Encodec VALL-E LLaMA classifier-free guidance generalized energy distance
Closest papers by meaning
- Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis (ICLR 2025 · similarity 0.88)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.87)
- DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.86)
- FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.86)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.86)
- SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.