Research map / Image generation / Autoregressive and latent generation

AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Diffusion audio synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AudioCALM, a continuous autoregressive LM with a rectified-flow head, block-causal AR-Flow attention, description-style conditioning and an asymmetric speech-only residual expert (A-MoME) to unify speech, sound, and music generation.

Problem

Unified audio generation across speech, sound, and music is hindered by fidelity loss from discrete codecs, lack of end-to-end optimization in cascaded systems, missing in-context conditioning in non-AR flow models, and cross-modal interference from mixing speech with non-speech training.

Approach

Extend next-token prediction to continuous VAE latents via a thin rectified-flow head on LM hidden states, use block-causal attention (causal across blocks, bidirectional within a block) for variable-length output, recast all captions as long-form descriptions, and add a speech-only residual FFN expert to address the local-vs-global text-audio mismatch.

Key results

• On LibriTTS test-clean, AudioCALM reaches WER 0.020 and MOS 4.02, the lowest WER and highest MOS among listed systems, while CosyVoice 3.0 keeps higher SIM (0.697 vs 0.668).
• On AudioCaps text-to-sound, AudioCALM lowers FAD to 1.95 from 2.70 for TangoFlux (28% relative reduction) and CLAP to 0.37.
• On Song-Describer, AudioCALM achieves FAD 2.02 and CLAP 0.36 versus Stable Audio Open's 2.23 and 0.32.

Surprising findings

• Adding speech to a non-speech-only training run degrades sound and music FAD by about 35%, whereas adding non-speech to speech-only training leaves speech metrics within noise.
• Continuous flow-matching head vs discrete codec head is the single largest contributor across all three domains in the ablation.

Limitations

• Training mixture limited to English speech and public sound and music corpora; non-English speech, singing voice, and some audio events uncovered.
• Gap to CosyVoice 3.0 on speaker similarity attributed to its larger English speech corpus and speaker-fidelity objective.
• Backbone size limited to 1.7B in main results; open questions around long-form generation and scale.

Experimental setup

Datasets: LibriTTS Emilia VGGSound AudioCaps WavCaps FMA MTG-Jamendo SeedTTS-eval Song-Describer

Benchmarks: LibriTTS test-clean SeedTTS-eval (en) AudioCaps Song-Describer

Metrics: WER SIM MOS FAD CLAP MOS-Q MOS-T

Baselines: F5-TTS CosyVoice 3.0 AudioLDM 2-Large TangoFlux Stable Audio Open MusicGen-Large UniAudio UniMoE-Audio UniFlow-Audio Ming-omni

Builds on: DiTAR rectified flow MAR Qwen3 Stable Audio Open VAE

Closest papers by meaning

• Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.92)
• UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.91)
• A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.90)
• Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.89)
• Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.89)
• UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.89)