Research map / Large language models / Language, safety and interpretability

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction

ICML 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a causal audio language model over continuous VAE latents with a per-token diffusion head, and adds masked next-token prediction (drop-based skip-token prediction with target positional embeddings) to reach near-SOTA text-to-audio on AudioCaps.

Problem

Autoregressive audio generation has relied on discrete tokens, which are long sequences and lag behind diffusion models in fidelity; the paper asks whether a causal LM on continuous audio tokens can reach high-fidelity text-to-audio generation.

Approach

Replace discrete audio tokens and cross-entropy with continuous VAE latents modeled by a token-wise diffusion head on a causal Transformer, then introduce masked next-token prediction that drops random tokens and predicts skip-future tokens via target positional embeddings, generalizing causal LM to predicting any future position given any subset of past tokens.

Key results

• AudioNTP Base (193M) gets about 20% relative FD and 40% relative KL gains over AudioGen Base (285M) on AudioCaps, per the paper's text (FAD 2.28 vs 2.84 is the table's FAD comparison).
• AudioMNTP Base reaches FAD 1.68 (41% relative improvement over AudioGen Base) and AudioMNTP Large (462M) reaches FAD 1.22 and FD 14.30, the best FD and FAD in the table.
• Human evaluation: AudioMNTP outperforms AudioGen on speech and non-speech REL and OVL, approaching Tango 2.

Measured effects reported in the paper

Surprising findings

• Dropping tokens (rather than zero-masking) with skip-token prediction improves causal next-token modeling; ablation shows removing the target positional embedding is worse than plain next-token prediction.

Limitations

• Trained only on AudioCaps and WavCaps (about 1,000 hours); scaling to larger data not explored.
• Unidirectional model is less expressive than bidirectional diffusion models; REL lags Tango 2.
• Results rely on the image MAR initialization for some gains, and masking schedule is sensitive to strategy.

Experimental setup

Datasets: AudioCaps WavCaps

Benchmarks: AudioCaps

Metrics: FD FAD KL IS CLAP score REL OVL

Baselines: AudioGen Base AudioGen Large UniAudio MAGNET-Small MAGNET-Large AudioLDM2-Full Tango 2 Make-an-Audio 2 AudioMAR

Builds on: MAR AudioLDM AudioGen classifier-free guidance

Closest papers by meaning

• AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.92)
• IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling (ICML 2025 · similarity 0.89)
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.89)
• A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.89)
• ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.89)
• Masked Audio Generation using a Single Non-Autoregressive Transformer (ICLR 2024 · similarity 0.89)