Research map / Large language models / Alignment and preferences
SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
Xueyao Zhang, Chaoren Wang, Huan Liao, Ziniu Li, Yuancheng Wang, Li Wang, Dongya Jia, Yuanzhe Chen, Xiulin LI, Zhuo Chen, Zhizheng Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use generative reward model (SFT+RL) instead of Bradley-Terry reward model | Qwen2.5-Omni-7B judge · SpeechJudge-Eval | accuracy: 72.7 -> 77.2 | Improved |
| Add GRPO RL stage after SFT (ablation) | SpeechJudge-GRM · SpeechJudge-Eval | accuracy: 75.3 -> 77.2 | Improved |
| Add majority voting over 10 outputs (ablation) | SpeechJudge-GRM (SFT) · SpeechJudge-Eval | accuracy: 75.3 -> 77.6 | Improved |
| Add majority voting over 10 outputs (ablation) | SpeechJudge-GRM (SFT+RL) · SpeechJudge-Eval | accuracy: 77.2 -> 79.4 | Improved |
| Use CoT prompt instead of plain prompt (ablation) | Gemini-2.5-Flash · SpeechJudge-Eval | accuracy: 69.1 -> 70.5 | Improved |
| Use CoT prompt instead of plain prompt (ablation) | Qwen2.5-Omni-7B · SpeechJudge-Eval | accuracy: 60.6 -> 52.9 | Hurt |
| Use CoT prompt instead of plain prompt (ablation) | Kimi-Audio-7B-Instruct · SpeechJudge-Eval | accuracy: 67.0 -> 66.5 | No clear effect |
| Add 2-shot exemplars to the prompt (ablation) | Qwen2.5-Omni-7B · SpeechJudge-Eval | accuracy: 60.6 -> 48.2 | Hurt |
| Add GRPO RL stage after SFT (ablation) | SpeechJudge-GRM · Human vs SeedTTS clone (OOD) | accuracy: 40.8 -> 62.4 | Improved |
| Use SpeechJudge-GRM (SFT+RL) instead of BTRM | Qwen2.5-Omni-7B judge · Human vs SeedTTS clone (OOD) | accuracy: 50.4 -> 62.4 | Improved |
| Post-train Qwen2.5-0.5B-TTS with SpeechJudge-GRM reward instead of INTP DPO data | Qwen2.5-0.5B-TTS · SeedTTS-Eval and Amphion-TTS-Eval | naturalness CMOS | Improved |
Closest papers by meaning
- SpeechAlign: Aligning Speech Generation to Human Preferences (NeurIPS 2024 · similarity 0.89)
- Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.89)
- Same Words, Different Judgments: How Preferences Vary Across Modalities (NeurIPS 2026 · similarity 0.88)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.88)
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.88)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
