Research map / Image generation / Autoregressive and latent generation
Continuous Audio Language Models
Simon Rouard, Manu Orsini, Axel Roebel, Neil Zeghidour, Alexandre Défossez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace 8-RVQ RQ-Transformer head with 1-step consistency head | Speech continuation LM · Speech continuation | Acoustic quality MOS (temp 0.8): 2.75 -> 3.45 | Improved |
| Lower Gaussian sampling temperature from 1.0 to 0.8 (ablation) | CALM speech continuation · Speech continuation | PPX: 42.9 -> 23.8 | Improved |
| Add noise augmentation to the backbone context (ablation) | CALM music, consistency 4 steps · Music continuation ablation | FAD: 1.63 -> 0.93 | Improved |
| Add short-context transformer (ablation) | CALM music, consistency 4 steps · Music continuation ablation | FAD: 4.03 -> 0.93 | Improved |
| Add head batch multiplier (ablation) | CALM music, consistency 4 steps · Music continuation ablation | FAD: 1.32 -> 0.93 | Improved |
| Use consistency head 1 step instead of RQ-Transformer 32 RVQ | Music continuation LM · Music continuation | FAD: 1.06 -> 0.83 | Improved |
| Use 4 consistency steps instead of 1 (ablation) | CALM music · Music continuation | FAD: 0.83 -> 0.71 | Improved |
| Use TrigFlow 100-step head instead of consistency (ablation) | CALM music · Music continuation | FAD / speed: FAD 0.64 vs 0.71; overall speedup x0.3 vs x1.9 | Mixed |
| Add short-context transformer and noise injection for speech (ablation) | CALM speech continuation · Speech continuation | quality | No clear effect |
| Reduce VAE to 32 dims for music (ablation) | Music VAE · Music compression | VisQOL: 4.01 (128 dims) -> 2.23 (32 dims) | Hurt |
| Use continuous VAE (32 dims) instead of 8-RVQ Mimi codec for speech | Speech codec · Speech compression | PESQ: 2.13 -> 2.42 | Improved |
| CALM TTS vs DSM on speaker similarity | CALM TTS with LSD · LibriSpeech test-clean | WavLM speaker similarity: 0.67 -> 0.52 | Hurt |
Closest papers by meaning
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.91)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.90)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.90)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.89)
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.88)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
