Research map / Large language models / Language, safety and interpretability
Speech and audio language models
Combine large language models with speech and audio understanding and generation.
111 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Language, safety and interpretability, keyword cluster “natural language · speech · generation”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
20 papers in 2023–24 and 84 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 2.9% to 4.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Toward a Better Understanding of Deep Neural Network Based Acoustic Modelling: An Empirical Investigation (AAAI 2016 · 1 citation)
- 2021 · DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances (AAAI 2021 · 69 citations)
- 2022 · Towards Building ASR Systems for the Next Billion Users (AAAI 2022 · 50 citations)
- 2023 · HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models (NeurIPS 2023 · 14 citations)
- 2024 · AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (AAAI 2024 · 114 citations)
- 2025 · Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · 20 citations)
- 2026 · MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence (AAAI 2026 · 1 citation)
Earliest papers here
- Toward a Better Understanding of Deep Neural Network Based Acoustic Modelling: An Empirical Investigation (AAAI 2016 · 1 citation)
- DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances (AAAI 2021 · 69 citations)
- Synchronous Interactive Decoding for Multilingual Neural Machine Translation (AAAI 2021 · 13 citations)
What has been tried
63 papers here report 720 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 5%, hurt 26%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Increase X-Codec quantization layers from n_q=1 to n_q=8 | Phonetic ABX within error rate · ABX error rate | 3.9 -> 3.3 | Helped | Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · 20 citations) |
| Add an SER emotion label | test-emo · GPT-4o judge score | 5.30 -> 6.11 | Helped | SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · 10 citations) |
| Train the speaking backbone from scratch and fix the pre-trained listening encoder rather than continuing training both | Command-based FDM · WER (%) | 4.05 -> 4.82 | Hurt | Language Model Can Listen While Speaking (AAAI 2025 · 8 citations) |
| Use 1x, 2x, or 3x synthetic dialogues per ground-truth dialogue | Primock57 ASR-transcribed test set · F1 | 20.93; 22.84; 22.51 | Mixed | MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues (AAAI 2025 · 6 citations) |
| Add an LLM rather than use CTC without an LM | Librispeech and GigaSpeech · WER (%) | 2.66 -> 1.96; 4.97 -> 4.18; 12.77 -> 10.63; 12.77 -> 11.05 | Helped | Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · 6 citations) |
| Fine-tune BEATs instead of freezing it | watkins · accuracy | 0.490 -> 0.723 | Helped | NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics (ICLR 2025 · 6 citations) |
| Use decoder-only rather than encoder-decoder architecture | EMIME Chinese→English · ASR-BLEU | 16.8 -> 20.7 | Helped | PolyVoice: Language Models for Speech to Speech Translation (ICLR 2024 · 5 citations) |
| decode with chunk=4 instead of chunk=∞ | AISHELL-1 · CER (%) | 2.15 -> 2.79 | Hurt | Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM (ICML 2025 · 4 citations) |
| Increase ICL examples from 8 to 16 | CREMA-D, Ravdess, US8K, GTZAN, Medley-solos-DB · ACC | Mixed | Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities (ICML 2024 · 4 citations) | |
| Replace GPT-4o with OpenAI o1 as the caption reasoner | MMAR · accuracy (%) | 50.70 -> 53.00 | Helped | MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · 3 citations) |
| add utterance-level language-space embedding only | CHiME-4 avg. · WER (%) | 6.9 -> 6.7 | Helped | Large Language Models are Efficient Learners of Noise-Robust Speech Recognition (ICLR 2024 · 3 citations) |
| use multi-scale 3-layer RVQ instead of vanilla 3-layer RVQ | ESC50 sound event classification, 2-way 1-shot · Accuracy (%) | 55 -> 60 | Helped | UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner (NeurIPS 2024 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 50 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- AudioSet · 11 of 50 papers
- LibriSpeech · 10 of 50 papers
- AudioCaps · 9 of 50 papers
- LibriTTS · 7 of 50 papers
- WavCaps · 6 of 50 papers
- FSD50K · 4 of 50 papers
- VCTK · 4 of 50 papers
- CommonVoice · 3 of 50 papers
Baselines
- SALMONN · 11 of 50 papers
- GAMA · 6 of 50 papers
- Qwen-Audio-Chat · 5 of 50 papers
- Qwen2.5-Omni · 5 of 50 papers
- Qwen2-Audio-Instruct · 5 of 50 papers
- EnCodec · 4 of 50 papers
- DAC · 4 of 50 papers
- Qwen-Audio · 4 of 50 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 104 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 9 papers
- Amazon · 6 papers
- Zhejiang University · 6 papers
- ByteDance · 6 papers
- Peking University · 5 papers
- Microsoft · 5 papers
- NVIDIA · 5 papers
- Nanyang Technological University · 5 papers
- Shanghai Jiao Tong University · 4 papers
- National University of Singapore · 4 papers
Use it
Download BibTeX of 26 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Text generation and evaluation metrics (79 papers)
- Multilingual LLM data and culture (69 papers)
- Sign language datasets and translation (23 papers)
- Sign and cross-lingual translation (96 papers)
Elsewhere in the topic
- LLM unlearning and benchmarks (298 papers)
- LLM pruning and scaling data (285 papers)
- Controlled text generation (220 papers)
- Faithfulness of LLM explanations (169 papers)
- Data and evaluation methodology (136 papers)
- Uncertainty quantification for LLMs (136 papers)
- Pruning and circuits in LLMs (135 papers)
- Statistical frameworks for LLM evaluation (124 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).