Research map / Large language models / Language, safety and interpretability

A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Speech and audio language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReasoningCodec, a discrete audio tokenizer factorizing audio into text-aligned reasoning tokens and reconstruction tokens, and builds UniAudio 2.0, a unified autoregressive audio-text model (3B LLaMA-3.2 backbone) trained with multi-stage training on 100B text and 60B audio tokens.

Problem

Audio language models lack a discrete audio tokenizer that serves both understanding (text-aligned, high-level information) and high-fidelity generation, and existing unified models lack few-shot/zero-shot generalization comparable to text LLMs.

Approach

Factorize audio into two role-separated discrete streams: reasoning tokens (5 Hz, trained via SFT and GRPO to be text-aligned for an LLM head) and reconstruction tokens (multi-expert semantic VQ with FiLM conditioning on reasoning tokens, decoded by a flow-based latent diffusion decoder). The backbone splits layers into audio-understanding, cross-modal (LLM-initialized), and audio-generation exper

Key results

• ReasoningCodec reconstruction: speech PESQ WB 2.36 and UT-MOS 3.91 vs best baseline 2.20 and 3.90 (Higgs-Audio); MUSHRA speech 90.5 vs 84.4 for Higgs-Audio at 100 TPS.
• UniAudio 2.0 achieves ASR WER of 2.7 on LibriSpeech test-clean, outperforming MiMo-Audio-7B-Instruct (3.5) with 3B parameters.
• Few-shot emotion classification (EN) 1-shot accuracy 67.0% vs 42.5% for MiMo-Audio; zero-shot dysarthric speech recognition WER 19.4% vs 80.6% for Qwen2.5-Omni.

Surprising findings

• Reasoning tokens alone achieve lower LLM perplexity (11.57 avg speech) than the full reconstruction tokens, and combining them lowers reconstruction-token perplexity further.
• UniAudio 2.0 is weaker than Qwen2.5-Omni-7B on some tasks such as TTS ZH/EN and LibriSpeech-other ASR, and its S2S score is below LLAMA-Omni.

Limitations

• Zero-shot tasks such as speech-to-speech and speech-sound generation are not solved perfectly, per the authors.
• Context length limited to 2048, preventing the same few-shot protocol as MiMo-Audio.
• Scaling results indicate the 1B variant is substantially weaker, suggesting capacity limits at small sizes.

Experimental setup

Datasets: Multilingual LibriSpeech AudioSet Million Song Dataset WenetSpeech IEMOCAP AudioCaps WavCaps LibriSpeech TUT Acoustic scenes 2017 LP-MusicCaps VCTK MusicCaps

Benchmarks: LibriSpeech test-clean/other Seed-TTS EN/ZH AudioCaps MusicCaps MMLU InstructS2S-Eval DNS-MOS MUSHRA ASR emotion recognition audio classification music classification

Metrics: PESQ STOI UTMOS VISQOL SIM AudioBox score WER CER KL FD

Baselines: DAC EnCodec MimiCodec Higgs-Audio tokenizer X-Codec ALMTokenizer MiMo-Audio-7B-Instruct Qwen2.5-Omni-7B MusicGen Stable Audio Open SongGen CapSpeech-AR

Builds on: GRPO FiLM SimpleSpeech UniAudio Qwen-Audio

Closest papers by meaning

• AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.90)
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.90)
• ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.90)
• MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.89)
• Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.89)
• UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner (NeurIPS 2024 · similarity 0.89)