Research map / Large language models / Language, safety and interpretability
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SD-Eval, a 7,303-utterance speech benchmark for evaluating spoken dialogue responses conditioned on emotion, accent, age, and background sound, plus a 1,052-hour training set and experiments with cascade and end-to-end speech LLMs judged by LLM-based metrics.
Problem
Speech LLMs can recognize and analyze speech but often fail to generate appropriate responses that account for paralinguistic and environmental information, and there are no open benchmarks or suitable metrics to evaluate this.
Approach
Build a benchmark from eight public speech datasets (plus synthesized speech) covering four paralinguistic/environmental dimensions, generate reference responses with GPT-4o, and propose reference-free LLM-judge scoring that correlates better with human judgment than BLEU/ROUGE/METEOR/BERTScore.
Key results
- On test-emo, VS-LLM (speech input with paralinguistic info) reaches BLEU-4 8.29 and GPT-4o judge 5.30 vs Cascade LLM 4.66 and 4.47, with the LLM upper bound at 12.35 and 6.74.
- LLM judges correlate far better with human evaluation than n-gram or embedding metrics; GPT-4o reaches Spearman 0.613 overall vs 0.291 for BERTScore and 0.208 for BLEU-4.
- Open-source speech LLMs score poorly on SD-Eval; e.g., Qwen-Audio gets GPT-4o judge scores of 1.72-2.50 across subsets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use direct speech input instead of ASR transcript input | VS-LLM versus Cascade LLM · test-emo, test-acc, test-age, test-env | GPT-4o judge score: 4.47 -> 5.30; 6.62 -> 7.85; 4.47 -> 7.95; 5.66 -> 5.82 | Helped |
| Use direct speech input instead of ASR transcript input | VS-LLM versus Cascade LLM · test-emo, test-acc, test-age, test-env | Human evaluation: 5.05 -> 6.31; 6.71 -> 7.95; 6.51 -> 7.11; 6.62 -> 7.11 | Helped |
| Provide ground-truth transcripts and labels rather than direct speech input | LLM (Upper Bound) versus VS-LLM · test-emo, test-acc, test-age, test-env | GPT-4o judge score: 5.30 -> 6.74; 7.85 -> 8.02; 7.95 -> 8.25; 5.82 -> 7.40 | Helped |
| Add an SER emotion label (ablation) | Speech LLM without text input or emotion label · test-emo | GPT-4o judge score: 5.30 -> 6.11 | Helped |
| Use a ground-truth emotion label rather than an SER label (ablation) | Speech LLM · test-emo | GPT-4o judge score: 6.11 -> 6.41 | Helped |
| Add an SER emotion label (ablation) | LLM with ASR transcript · test-emo | GPT-4o judge score: 4.47 -> 6.13 | Helped |
| Replace the ASR transcript with a ground-truth transcript while retaining the SER label (ablation) | LLM with SER emotion label · test-emo | GPT-4o judge score: 6.13 -> 6.38 | Helped |
| Replace the SER emotion label with a ground-truth label while retaining the ASR transcript (ablation) | LLM with ASR transcript · test-emo | GPT-4o judge score: 6.13 -> 6.47 | Helped |
| Replace the ASR transcript with a ground-truth transcript while retaining the ground-truth emotion label (ablation) | LLM with ground-truth emotion label · test-emo | GPT-4o judge score: 6.47 -> 6.74 | Helped |
| Use LLM-based judges rather than traditional or embedding-based metrics | Evaluation metrics on SD-Eval · Overall | Spearman correlation with human evaluation: BLEU-4: 0.208; ROUGE-L: 0.258; METEOR: 0.336; BERTScore: 0.291; GPT-4o: 0.613 | Helped |
Surprising findings
- Open-source speech LLMs such as SALMONN score higher on some LLM-judge metrics than Qwen2-Audio analysis mode, and the Qwen2-Audio voice-chat mode performs much better than the other open models.
- Reference-based metrics like BLEU and ROUGE show weak correlation with human judgment (Spearman about 0.2), while LLM judges reach about 0.6.
Limitations
- Only speech-to-text, so responses are evaluated at text level only.
- Single-turn dialogues only.
- Does not cover other paralinguistic aspects such as speaker gender.
Experimental setup
Datasets: RAVDESS MEAD JL Corpus VCTK Common Voice LibriSpeech AudioCaps MyST MSP-Podcast IEMOCAP MELD EmoV-DB
Benchmarks: SD-Eval
Metrics: BLEU-4 ROUGE-L METEOR BERTScore LLM judge score (1-10) human evaluation score Spearman correlation Kendall-Tau correlation
Baselines: SALMONN Qwen-Audio Qwen2-Audio Cascade LLM LLM (Upper Bound)
Builds on: LoRA Whisper xtuner BERTScore GPTScore
Closest papers by meaning
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.89)
- SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation? (NeurIPS 2026 · similarity 0.89)
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.89)
- Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.89)
- Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.88)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.