Research map / Large language models / Language, safety and interpretability
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Expands the Dynamic-SUPERB benchmark to 180 community-contributed instruction-based speech, music, and audio tasks, and evaluates several universal audio-language models using GPT-4o as an LLM judge and post-processor.
Problem
Existing benchmarks for instruction-following universal speech/audio models are small and limited to classification tasks, with no comprehensive coverage of speech, music, and audio across regression and generation.
Approach
A community-driven, continually expanding benchmark where all tasks are text-instructed test-only sets, outputs are free-form natural language judged by an LLM (GPT-4o) for classification, and a post-processing LLM converts outputs for regression, with N/A rates reported, plus a fine-grained task taxonomy.
Key results
- SALMONN-13B achieved the best English ASR result with WER below 10% (2.8%), and Qwen2-Audio-7B-Instruct reached about 75.8% accuracy on emotion recognition.
- No model excelled universally; speaker verification results for all models were close to random (about 45-51% accuracy), and query-by-example stayed near chance.
- Spoken language models trained mainly on speech (Qwen, SALMONN, WavLLM) outperformed music-specific models GAMA-IT and Mu-LLaMA in several music domains such as harmony and pitch and rhythm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SALMONN instead of other evaluated models | Models evaluated on SUPERB phoneme recognition · SUPERB Phoneme Recognition | PER: 100.1 -> 24.6 | Helped |
| Use Qwen-Audio-Chat instead of the other evaluated models | Models evaluated on SUPERB keyword spotting · SUPERB Keyword Spotting | accuracy: 36.5 -> 60.5 | Helped |
| Use Whisper-LLaMA instead of the other evaluated models | Models evaluated on SUPERB intent classification · SUPERB Intent Classification | accuracy: 36.5 -> 36.5 | Helped |
| Use Qwen2-Audio-7B-Instruct instead of other evaluated models | Models evaluated on SUPERB emotion recognition · SUPERB Emotion Recognition | accuracy: 12.5 -> 75.8 | Helped |
| Use SALMONN-13B instead of the other evaluated models | Models evaluated on SUPERB ASR · SUPERB ASR | WER: 34.0 -> 2.8 | Helped |
| Use evaluated models instead of relying on them for query-by-example | Models evaluated on SUPERB query-by-example · SUPERB Query-by-Example | accuracy: 46.5 -> 53.5 | No clear effect |
| Use SALMONN-7B instead of the other evaluated models | Models evaluated on SUPERB slot filling · SUPERB Slot Filling | F1: 53.6 -> 61.6 | Helped |
| Use Whisper-LLaMA instead of the other evaluated models | Models evaluated on SUPERB slot filling · SUPERB Slot Filling | CER: 51.3 | Helped |
| Use evaluated models instead of expecting reliable speaker verification | Models evaluated on SUPERB speaker verification · SUPERB Speaker Verification | accuracy: 45.0 -> 51.0 | No clear effect |
| Use speech models rather than Whisper-LLaMA for speaker and paralinguistics tasks | Models evaluated across speech domains · Speaker and paralinguistics domains | relative score | Helped |
| Train spoken language models on diverse data spanning speech, music, and audio | Spoken language models · Music-related domains | relative score | Helped |
| Use SALMONN rather than Whisper-LLaMA for phoneme recognition | Models evaluated across speech domains · Phoneme recognition | PER | Helped |
Surprising findings
- Cascaded Whisper-LLaMA outperformed all end-to-end models on speech recognition and spoken language understanding domains at the domain level.
- Speech-trained spoken language models outperformed dedicated music models on several music tasks.
Limitations
- Lacks speech generation tasks; Phase-2 focuses on understanding tasks.
- LLM-based evaluation may not generalize to all future tasks, though it correlates with humans.
- Results depend on GPT-4o, which may change over time; open-source judge results are provided as a mitigation.
Experimental setup
Datasets: LibriSpeech SUPERB tasks MARBLE tasks HEAR tasks ESC50 MTG-Jamendo SLURP MELD VoxAngeles speechocean762 MUSAN VCTK
Benchmarks: Dynamic-SUPERB Phase-2
Metrics: accuracy word error rate phoneme error rate F1 character error rate diarization error rate mean square error N/A rate relative score
Baselines: Whisper-LLaMA SALMONN Qwen-Audio-Chat Qwen2-Audio-7B-Instruct WavLLM LTU-AS GAMA-IT Mu-LLaMA
Builds on: Dynamic-SUPERB SUPERB MARBLE HEAR
Closest papers by meaning
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.89)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.88)
- Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.88)
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.87)
- Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.87)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.