Research map / Vision-language and multimodal / Multimodal LLMs

MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MECAT, a ~20k-clip multi-expert, CoT-LLM-annotated benchmark for fine-grained audio captioning and open-set QA, plus the DATE metric that combines embedding similarity with cross-sample discriminability, and evaluates 15+ audio LLMs.

Problem

Existing audio captioning and audio QA benchmarks use coarse, event-level or close-ended annotations and lexical or embedding metrics that cannot separate generic from detailed model outputs.

Approach

Build captions and open-set QA by fusing outputs from specialized expert models (speech, music, sound, acoustic quality) with DeepSeek-R1 chain-of-thought synthesis, and score responses with DATE, which weights TF-IDF Sentence-BERT similarity and rewards candidates that rank highest for their own reference among all samples (harmonic mean).

Key results

• Gemini-3-Pro scores highest on MECAT-Caption (Score_Cap 53.1), with Qwen3-Omni-Flash-1201 at 52.9 and Gemini-2.5-Flash at 51.6.
• On MECAT-QA, Qwen3-Omni (52.3) edges out Gemini-2.5-Flash (52.1); no model exceeds 40 on Quality Assessment except the top-tier range noted.
• Human A/B test (N=150): MECAT references win 82.7% overall and 94.2%+ against safe and wrong captions, and tie with human references at 56.9%.

Surprising findings

• Many LALMs hallucinate specific unrelated text (e.g., 'Thank you') on silent clips.
• Open-weight Qwen3-Omni (40.4 on captions) trails proprietary models on captioning but is on par on QA (52.3), narrowing the proprietary-open gap.

Limitations

• Clips limited to 10 seconds, so long-range temporal reasoning is not assessed.
• Residual annotation noise from expert model hallucinations, especially in non-speech categories.
• DATE inherits embedding-similarity weaknesses (e.g., insensitivity to entity swaps) and its discriminability weakens on homogeneous corpora.

Experimental setup

Datasets: ACAV100M AudioSet

Benchmarks: MECAT-Caption MECAT-QA

Metrics: DATE Score_Cap Score_QA Kendall's tau CDF separation Wilson CI win rate REL score

Baselines: Pengi EnClap FENSE BLEU-1 CIDEr SPICE LLM-as-judge

Builds on: Sentence-BERT FENSE AudioSet Audio Flamingo 2

Closest papers by meaning

• Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.90)
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.89)
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.89)
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.89)
• Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
• Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.89)