Research map / Large language models / Language, safety and interpretability
UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM-Codec, a semantic-guided multi-scale RVQ audio codec that maps audio into LLaMA 2's token vocabulary, enabling frozen LLaMA 2 7B to perform few-shot audio understanding and generation via in-context learning (UniAudio 1.5).
Problem
Frozen LLMs cannot directly handle audio tasks without fine-tuning; existing audio-LLMs need task-specific training data and do not support in-context learning of unseen audio tasks or audio generation.
Approach
Quantize audio with a codec whose first RVQ layer is initialized from LLaMA 2 word-level vocabulary embeddings (Oxford 5000 words) and trained with a semantic loss and a Whisper-based consistency loss, so audio becomes a "foreign language" in the LLM's token space that a frozen LLM can learn from a few demonstrations.
Key results
- On 2-way speech emotion classification (ESD), UniAudio 1.5 with semantic layer reaches 59% at 3-shot versus 50% for BLSP at 3-shot, with random at 44%.
- LLM-Codec compresses audio to 57 tokens per second at PESQ 2.55 and STOI 0.82 on VCTK, versus 99 tokens/s and PESQ 2.64 for the vanilla 3-RVQ baseline.
- Speech denoising reaches PESQ 2.17 and STOI 0.57 versus 3.53 and 0.79 for SGMSE+, and text-to-speech digit-query accuracy is 70%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LLM-Codec instead of vanilla 3-layer RVQ | Audio codec reconstruction and sound event classification · VCTK reconstruction | PESQ: 2.64 -> 2.55 | Hurt |
| use multi-scale 3-layer RVQ instead of vanilla 3-layer RVQ (ablation) | LLM-Codec · ESC50 sound event classification, 2-way 1-shot | Accuracy (%): 55 -> 60 | Helped |
| use multi-scale 3-layer RVQ instead of vanilla 3-layer RVQ (ablation) | LLM-Codec · VCTK reconstruction | PESQ: 2.64 -> 2.55 | Hurt |
| use down-sampling 320 instead of 480 (ablation) | LLM-Codec multi-scale 3 RVQ · VCTK reconstruction | PESQ: 2.55 -> 2.60 | Helped |
| use down-sampling 320 instead of 480 (ablation) | LLM-Codec multi-scale 3 RVQ · ESC50 sound event classification, 2-way 1-shot | Accuracy (%): 60 -> 57 | Hurt |
| add semantic loss (ablation) | LLM-Codec · ESC50 sound event classification, 2-way 1-shot | Accuracy (%): 58 -> 60 | Helped |
| add consistency loss (ablation) | LLM-Codec · VCTK reconstruction | PESQ: 1.19 -> 2.55 | Helped |
| add consistency loss (ablation) | LLM-Codec · ESC50 sound event classification, 2-way 1-shot | Accuracy (%): 48 -> 60 | Helped |
| use word-level codebook instead of sub-word codebook (ablation) | LLM-Codec first VQ layer · VCTK reconstruction | PESQ: 2.46 -> 2.55 | Helped |
| use word-level codebook instead of sub-word codebook (ablation) | LLM-Codec first VQ layer · ESC50 sound event classification, 2-way 1-shot | Accuracy (%): 59 -> 60 | Helped |
| update codebooks instead of keeping them frozen (ablation) | LLM-Codec · VCTK reconstruction | PESQ: 2.55 -> 2.63 | Helped |
| update codebooks instead of keeping them frozen (ablation) | LLM-Codec · ESC50 sound event classification, 2-way 1-shot | Accuracy (%): 60 -> 55 | Hurt |
Surprising findings
- Repeating demonstrations lowered BLSP accuracy, while UniAudio 1.5 benefited from repeats.
- Updating the codebooks improved reconstruction but dropped downstream classification accuracy, supporting the frozen LLM-vocabulary codebook design.
Limitations
- Performance on in-context audio tasks remains well below specialized audio models.
- Limited by LLM context length, so only a few demonstrations can be added.
- Only LLaMA 2 7B was explored as the backbone; the codec is trained on English-centric data.
Experimental setup
Datasets: MLS AudioCaps ESD ESC-50 VCTK NoiseX-92 Free Spoken Digit Dataset Speech Commands Dynamic-SUPERB
Benchmarks: ESD ESC-50 Dynamic-SUPERB FSDD VCTK Speech Commands
Metrics: accuracy PESQ STOI DNSMOS
Baselines: BLSP ImageBind-LLM Whisper-LLM ASR-ChatGPT Encodec DAC SGMSE+ FastSpeech 2
Builds on: VQ-VAE EnCodec DAC SpeechTokenizer UniAudio
Closest papers by meaning
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.90)
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.89)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.89)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.87)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.87)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.