Research map / Large language models / Language, safety and interpretability

UniAudio: Towards Universal Audio Generation with Large Language Models

ICML 2024 · Large language models › Language, safety and interpretability · approach: Speech and audio language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniAudio, a ~1B-parameter LLM-based model trained on ~100k hours of public audio that unifies 11 audio generation tasks (speech, sound, music, singing) via discrete multimodal tokens and a hierarchical CodecFormer architecture.

Problem

Audio generation research is dominated by task-specific models with heavy domain knowledge; LLM-based audio models typically handle only one or a few tasks, and long RVQ-based audio token sequences make LLM modeling costly.

Approach

Tokenize all modalities (audio via a custom 3-RVQ codec, phonemes, MIDI, semantic tokens, T5 text embeddings) into one sequence format and model them with CodecFormer, a hierarchical patch predictor plus token predictor that treats each RVQ frame as a patch to avoid T×N sequence lengths while keeping flattened autoregression.

Key results

• On the 7 training-stage tasks, UniAudio beats the chosen prior work in 3 of 6 subjective comparisons and 5 of 7 objective comparisons, but underperforms on Text-to-Music versus Copet et al. (2023).
• After fine-tuning on 4 new tasks, UniAudio surpasses baselines on audio edit (FD 17.78 vs 20.78) and speech dereverberation (PESQ 2.13 vs 2.87 is lower, DNSMOS 3.51 vs 3.42 is higher), and approaches ground truth on prompt TTS.
• Ablation shows jointly trained UniAudio outperforms single-task models on all 11 tasks; CodecFormer matches naive flattening quality (MCD 6.52 vs 6.56) at lower training cost (19.4 GB vs 36.7 GB GPU memory).

Measured effects reported in the paper

Surprising findings

• Adding out-of-domain sound and speech data improved text-to-music generation over music-only training.
• Signal-level metrics such as PESQ and VISQOL may not be suitable for LLM-based generative methods, so UniAudio looks weaker on them despite competitive subjective scores.

Limitations

• UniAudio underperforms the selected competitor on Text-to-Music, which the authors attribute to that competitor using more private labeled data.
• Signal-level metrics show weaker results on several tasks, and the authors note these may not suit LLM-based generation.
• Fine-tuned tasks have less data, and the model's performance on them is less competitive on some metrics such as SE and TSE subjective scores.

Experimental setup

Datasets: LibriTTS Million Song Dataset m4singer musiccaps VCTK clotho

Benchmarks: LibriTTS Million Song Dataset VCTK musiccaps m4singer

Metrics: WER SIM PESQ VISQOL DNSMOS MCD FAD KL FD MOS

Baselines: Shen et al. (2023) Wang et al. (2023f) Chen et al. (2022) Žmolíková et al. (2019) Liu et al. (2022) Ghosal et al. (2023) Copet et al. (2023) SGMSE+ AUDIT Encodec DAC

Builds on: SPEAR TTS VALL-E MusicGen Encodec DAC

Closest papers by meaning

• Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.90)
• UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.89)
• AudioGenX: Explainability on Text-to-Audio Generative Models (AAAI 2025 · similarity 0.89)
• AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.89)
• A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.88)
• UALM: Unified Audio Language Model for Understanding, Generation and Reasoning (ICLR 2026 · similarity 0.88)