Research map / Vision-language and multimodal / Multimodal LLMs

UALM: Unified Audio Language Model for Understanding, Generation and Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces UALM, a single decoder-only audio language model (initialized from Qwen2.5-7B) that unifies audio understanding, text-to-audio generation via codec tokens, and text reasoning, plus UALM-Reason, which adds multimodal reasoning (rich captions, dialogue, self-reflection) for audio generation.

Problem

Audio understanding and text-to-audio generation are treated as separate tasks, usually with different paradigms (autoregressive LLMs vs. diffusion), and audio reasoning is mostly text-only and confined to understanding. The paper asks whether one language model can unify understanding, generation,

Approach

Builds a decoder-only LM that predicts discrete X-codec audio tokens (8 RVQ levels, delay pattern) with CFG and DPO, removes an external text encoder by using BPE tokens from a pretrained LLM, scales generation data to 30M pairs, and uses a modality-alignment warm-up before unifying with AF3-style understanding data. UALM-Reason then post-trains with interleaved SFT-DPO to produce rich captions, d

Key results

• UALM-Gen reaches FD 74.43 on SongDescriber and 75.14 on AudioCaps, outperforming the listed diffusion baselines on FD (e.g., ETTA 95.66 and 80.13).
• UALM scores 74.1 mean on MMAU (v05.15.25), best among listed open models, while scoring 55.2 on MMAR versus 58.5 for Audio Flamingo 3.
• UALM-Reason gets subjective 4.01 (enrichment), 4.02 (dialogue), and 4.04 (self-reflection) versus 3.77, 3.92, 3.82 for base UALM.

Surprising findings

• Text-to-audio LM generation needed about an order of magnitude more data than diffusion models; at 1/32 of the data it overfit.
• UALM text ability drops only slightly (MMLU 71.6 vs 74.5 for Qwen2.5-7B-Instruct), with GSM8K slightly higher (92.1 vs 91.6).

Limitations

• Synthetic captions from captioning models contain some hallucination and audio-caption misalignment.
• Lack of accurate automatic metrics for audio quality, aesthetics, and reasoning chains limits evaluation and RL.
• UALM-Reason's generative reasoning is evaluated mainly qualitatively and with small subjective tests (20 prompts per category).

Experimental setup

Datasets: AudioCaps SongDescriber AF3 understanding mixture

Benchmarks: AudioCaps SongDescriber MMAU MMAR MMLU GSM8K HumanEval

Metrics: Frechet distance KL divergence Inception Score CLAP score AudioBox-Aesthetic OVL REL accuracy

Baselines: MusicGen-stereo-L AudioGen-M MAGNeT-M AudioLDM2-L TangoFlux Stable Audio Open ETTA Audio Flamingo 3 Qwen2.5-Omni Qwen2.5-7B-Instruct

Builds on: MusicGen delay pattern Direct Preference Optimization Classifier-free guidance Audio Flamingo 3 Qwen2.5-7B X-codec

Closest papers by meaning

• Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.91)
• Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.89)
• UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.88)
• AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound (NeurIPS 2025 · similarity 0.88)
• MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.88)
• When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? (AAAI 2026 · similarity 0.88)