Research map / Large language models / Language, safety and interpretability
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AudioGPT, a system that pairs ChatGPT with existing audio foundation models plus ASR/TTS interfaces to handle speech, music, sound, and talking-head understanding and generation via spoken dialogue, without training a multimodal LLM.
Problem
Current LLMs cannot process complex audio information or hold spoken conversations, and training audio-capable multimodal LLMs from scratch is data- and compute-intensive.
Approach
Instead of training a spoken or multimodal LLM, use ChatGPT as an interface that analyzes user intent, assigns audio foundation models to tasks via structured arguments (prosody, timbre, language), and wraps the system with ASR/TTS for speech dialogue.
Key results
- AudioGPT solves 16 audio understanding and generation tasks across speech, music, sound and talking head in multi-round dialogue (qualitative case study, Figure 1).
- Experiments run gpt-3.5-turbo via LangChain with temperature zero and max 2048 tokens, with audio models deployed on a single NVIDIA T4 GPU on Hugging Face Space.
Limitations
- Evaluation framework (consistency, capability, robustness) is outlined but the excerpt shows only qualitative case studies, no quantitative results.
- Relies on a closed API LLM (gpt-3.5-turbo) for task planning and on a fixed set of external foundation models.
Experimental setup
Builds on: ChatGPT LangChain HuggingGPT Visual ChatGPT
Closest papers by meaning
- Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.88)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.88)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.88)
- Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.88)
- UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.