Research map / Large language models / Language, safety and interpretability

VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VocalCoachBench, an expert-annotated benchmark (515 singing recordings, 12,051 atomic claims) for evaluating audio-language models on vocal coaching feedback, combining deterministic structured tasks with claim-based open-ended evaluation, and benchmarks 12 models.

Problem

Audio-language models are evaluated on recognizing and describing audio but not on producing expert-style analytic feedback that identifies problems and suggests corrections for singing performances.

Approach

Separates deterministic structured targets (same-song triplet ranking, ranked Top-3 issue labels, segment-level issue classification) from claim-based evaluation of free-form diagnosis and correction, avoiding scalar scores or single reference responses that pilot studies showed had low expert agreement.

Key results

• Qwen3.5-Omni Plus reaches 74.5% pairwise triplet-ranking accuracy, the best among evaluated models; Qwen2.5-Omni reaches 68.7% among open-weight main-protocol models.
• No model exceeds the majority baseline (62.4%) on fine-level Top-3 F1@3; best scores are 50.1% (Gemini 3 Flash Preview) and 44.6% (Fun-Audio-Chat).
• Strict diagnosis hit rates stay below 7% across all models, while hit rates (strict or coarse) range from 17% to 63%.

Surprising findings

• Scalar score elicitation for triplet ranking collapses to ties on many triplets and yields pairwise accuracy below random for closed-source models.
• Segment-level MiMo-Audio matches the majority baseline only by predicting vocalization for every segment.

Limitations

• English solo singing only, same-song subset limited to one song (Amazing Grace).
• Audio-only, one-shot evaluation; segment-grounded evaluation uses expert-provided segments.
• Claim-based evaluation measures reference-supported coverage and may under-credit valid feedback outside expert claims.

Experimental setup

Datasets: DAMP-S-AG PopBuTFy N20EMv2 LM-SSD NUS-48E MedleyDB MedleyVox MRSSing URSing GTSinger

Benchmarks: VocalCoachBench

Metrics: pairwise accuracy Kendall's tau exact permutation accuracy cycle rate fine-level F1@3 Top-1 accuracy nDCG@3 parent-level F1 any-match accuracy diagnosis hit rate

Baselines: Random baseline Majority baseline R1-AQA Qwen2.5-Omni Kimi-Audio MiMo-Audio Qwen3-Omni Fun-Audio-Chat GPT-Audio 1.5 Gemini 3 Flash Preview Qwen3.5-Omni Flash Qwen3.5-Omni Plus

Builds on: Qwen3-Omni Qwen2.5-Omni DAMP-S-AG

Closest papers by meaning

• GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks (NeurIPS 2024 · similarity 0.88)
• TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching (AAAI 2025 · similarity 0.86)
• AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency (NeurIPS 2026 · similarity 0.85)
• Responding to the Call: Exploring Automatic Music Composition Using a Knowledge-Enhanced Model (AAAI 2024 · similarity 0.85)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.85)
• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.85)