Research map / Large language models / Language, safety and interpretability

Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces free-form-text-to-unified-audio generation and PlanAudio, an autoregressive Qwen2.5-1.5B-based model that plans via continuous semantic latent CoT before generating codec tokens, evaluated on a new PlanAudio-Bench for speech+sound composites.

Problem

Generating unified audio where speech and environmental sounds are composited from a single unconstrained natural-language prompt, without explicit text rewriting or multi-encoder pipelines, and with scarce joint speech-sound annotations.

Approach

A monolithic LLM takes raw text prompts (no text encoder) and first autoregressively predicts a fixed-length sequence of continuous semantic latents (regressed onto pooled AudioFlamingo 3 encoder embeddings) as an implicit plan, then generates hierarchical codec tokens conditioned on prompt and plan.

Key results

• On PlanAudio-Bench composite, PlanAudio reaches FD_PANNs 8.52 vs 14.3 for the AudioLDM2 pipeline, and subjective ratings of 3.23-3.47 across quality, temporal, semantic and authenticity, the highest of all compared models.
• On AudioCaps sound generation, PlanAudio attains FD_PANNs 24.7 (best) and CLAP 0.19, while specialist Tango/Make-An-Audio retain better IS/KL, so PlanAudio is competitive but not uniformly best.
• On LibriTTS, PlanAudio achieves WER 0.11 and UTMOS 3.11, beating VoiceLDM-s and VoiceLDM-m on UTMOS (2.99) but trailing VoiceLDM-m on composite WER (0.09 vs 0.41).

Surprising findings

• Explicit natural-language CoT helps sound generation but its advantage diminishes in composite scenarios, and acoustic codec-based CoT performs worst overall.
• Disjoint sequential curricula cause catastrophic forgetting of sound and speech, yet composite-only training still improved single-scenario performance versus pure single-scenario training.

Limitations

• Speech WER and UTMOS trail VoiceLDM-m on composite data, attributed to codec reconstruction ceiling and training data realism.
• Evaluation relies on a 4,500-sample benchmark built from Gemini and Whisper annotations and a 50-sample subjective study.

Experimental setup

Datasets: AudioSet AudioCaps WavCaps LibriTTS PlanAudio-Bench

Benchmarks: PlanAudio-Bench AudioCaps LibriTTS

Metrics: FD_PANNs FD_PaSST KL_PaSST KL_PANNs IS CLAP WER UTMOS SCF subjective Likert ratings (quality, temporal, semantic, auth

Baselines: VoiceLDM-s VoiceLDM-m AudioLDM2 Make-An-Audio Tango PromptTTS++ Pipeline (two AudioLDM2 variants) Explicit CoT Acoustic CoT w/o CoT

Builds on: Qwen2.5 AudioCraft tokenizer delayed token interleaving pattern VoiceLDM

Closest papers by meaning

• UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.90)
• AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.89)
• ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.89)
• UALM: Unified Audio Language Model for Understanding, Generation and Reasoning (ICLR 2026 · similarity 0.89)
• UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.88)
• A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.88)