Research map / Large language models / Language, safety and interpretability
Audio Large Language Models Can Be Descriptive Speech Quality Evaluators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a natural-language speech quality evaluation corpus from NISQA human ratings via LLM generation and proposes ALLD, a token-level distillation alignment method that trains audio LLMs (Qwen2-Audio) to give descriptive MOS and A/B judgments.
Problem
Audio LLMs are unaware of speech quality because no natural-language speech quality evaluation datasets exist; existing MOS work is regression-only without descriptive analysis.
Approach
Generate descriptive MOS and A/B comparison responses with LLaMA-3.1-70B conditioned on human-rated NISQA metadata, then align an audio LLM to these responses via a DPO-style objective where a smaller expert LLM (Qwen-7B) serves as the reference model for token-level distillation.
Key results
- ALLD (2x data) on Qwen2-Audio reaches MSE 0.17, LCC 0.93, SRCC 0.93 on MOS prediction, outperforming Wav2vec2 (MSE 0.27, LCC 0.93 in-domain regression baseline with lower MSE).
- ALLD with joint training achieves 98.6% A/B test accuracy and BLEU 30.17 versus 95.6% without joint training.
- ALLD achieves BLEU 25.84 on MOS description, surpassing SALMONN and Qwen-Audio baselines' BLEU (25.49 and 23.52).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ALLD instead of full finetuning | Qwen2-Audio · NISQA MOS prediction | BLEU: 23.84 -> 25.22 | Helped |
| Generate the training set twice with modified demonstrations and adjusted inference temperature (ablation) | Qwen2-Audio ALLD · NISQA MOS prediction | MSE: 0.20 -> 0.17 | Helped |
| Use IA3 parameter-efficient finetuning instead of full finetuning (ablation) | Qwen2-Audio · NISQA MOS prediction | MSE: 0.21 -> 1.45 | Hurt |
| Use LoRA on the encoder and decoder instead of full finetuning (ablation) | Qwen2-Audio · NISQA MOS prediction | MSE: 0.21 -> 0.52 | Hurt |
| Train the encoder only instead of full finetuning (ablation) | Qwen2-Audio · NISQA MOS prediction | MSE: 0.21 -> 0.24 | Hurt |
| Train the decoder only instead of full finetuning (ablation) | Qwen2-Audio · NISQA MOS prediction | MSE: 0.21 -> 0.55 | Hurt |
| Add descriptive analysis to the fixed-template output (ablation) | Qwen2-Audio ALLD · NISQA MOS prediction | MSE: 0.20 -> 0.17 | Helped |
| Jointly train on MOS prediction and A/B test instead of A/B-only training (ablation) | Qwen2-Audio ALLD · A/B test | Accuracy (%): 95.6 -> 98.6 | Helped |
| Jointly train on MOS prediction and A/B test instead of A/B-only training (ablation) | Qwen2-Audio ALLD · MOS prediction | MSE: 0.20 -> 0.20 | No clear effect |
| Jointly train on MOS prediction and A/B test instead of A/B-only training (ablation) | Qwen2-Audio ALLD · MOS prediction | BLEU: 25.22 -> 26.08 | Helped |
| Use ALLD instead of Wav2vec2 | Speech quality evaluation · FOR unseen speech domain | MSE: 0.13 -> 0.10 | Helped |
| Use ALLD instead of Wav2vec2 | Speech quality evaluation · P501 unseen speech domain | MSE: 0.43 -> 0.19 | Helped |
Surprising findings
- A template-only MOS output trained on the same data (no descriptions) achieves comparable MSE (0.20), yet adding descriptive diversity lowers MSE to 0.17, suggesting description helps numeric prediction in a CoT-like manner.
- Zero-shot in-context learning of existing open audio LLMs (Qwen2-Audio, SALMONN, SpeechGPT, AnyGPT, SALM) fails on speech quality evaluation with widespread hallucinations.
Limitations
- Word-level quality degradation cannot be precisely described; SWD only detects synthetic words without explaining causes.
- Descriptive analysis's causal link to MOS estimation is not fully understood.
- Zero-shot ICL of existing audio LLMs fails, so warm-up SFT is required before sampling.
Experimental setup
Datasets: NISQA LibriSpeech CommonVoice Fair-Speech
Benchmarks: NISQA_TRAIN_SIM (in-domain) NISQA_VAL_LIVE NISQA_TEST_FOR NISQA_TEST_P501 LibriSpeech SWD
Metrics: LCC SRCC MSE BLEU accuracy
Baselines: CNN-SA-AP WavLM Wav2vec2 SALMONN Qwen-Audio Qwen2-Audio full fine-tuning
Builds on: DPO Qwen2-Audio NISQA VoiceCraft IA3
Closest papers by meaning
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.88)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.88)
- Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models (ICLR 2026 · similarity 0.88)
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.88)
- TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.87)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.