Research map / Large language models / Language, safety and interpretability
BRACE: A Benchmark for Robust Audio Caption Quality Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BRACE, a reference-free benchmark with BRACE-Main (human/machine caption pair comparison) and BRACE-Hallucination (LLM noun-substitution hallucinations) to evaluate CLAPScore-style audio caption metrics and large audio language models.
Problem
Reference-free evaluation of audio captions (e.g., CLAPScore) lacks systematic validation of robustness, and existing benchmarks do not test full-caption quality comparison or full-length caption hallucination detection in audio.
Approach
Build pairwise audio-caption comparison data from AudioCaps/Clotho evaluation sets (LALM-generated and LLM-corrupted captions, with three-annotator consensus filtering) plus LLM-generated noun-substitution hallucinations, and use it to meta-evaluate CLAP-based metrics and LALMs; also proposes a sliding-window CLAP embedding to reduce run-to-run variance.
Key results
- Best CLAP-based metric (SLIDE-CLAP LAION-CLAP) reaches 70.01 average F1 on BRACE-Main; best LALM AF2 reaches 63.19.
- On BRACE-Hallucination, GPT-4o-Audio-Preview attains 96.37 F1 while the best open-source LALM Qwen-Audio-Chat reaches 77.25.
- CLAP models reach no more than 70 on HH and MM pairs in BRACE-Main, indicating weak fine-grained caption discrimination.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply sliding-window audio embedding aggregation instead of standard CLAP audio-window handling (ablation) | CLAP models on BRACE-Main · BRACE-Main | F1-score: 68.93 -> 70.01 | Helped |
| Add reference captions to CLAP evaluation (ablation) | CLAP models · BRACE-Main Avg | F1-score: 71.54 -> 72.74 | Helped |
| Add reference captions to CLAP evaluation (ablation) | CLAP models · BRACE-Hallucination Avg | F1-score: 95.85 -> 97.92 | Helped |
| Use simple rather than naive prompt wording (ablation) | Qwen-Audio-Chat · BRACE-Main | F1-score: 28.99 -> 59.64; 17.69 -> 59.84 | Mixed |
| Use complex rather than naive prompt wording (ablation) | AF2 · BRACE-Main | F1-score: 62.94 -> 1.35; 63.19 -> 3.26 | Hurt |
| Provide a tie option in a naive prompt (ablation) | GPT-4o-Audio-Preview · BRACE-Main | F1-score: 68.30 -> 69.50 | Helped |
| Provide a tie option in a naive prompt (ablation) | AF2 · BRACE-Main | F1-score: 62.94 -> 63.19 | Helped |
| Add a single reference caption and use a simple tie prompt (ablation) | Qwen2-Audio-Instruct · BRACE-Main | F1-score: 52.73 -> 64.24 | Helped |
| Add a single reference caption and use a simple non-tie prompt (ablation) | Qwen2-Audio-Instruct · BRACE-Hallucination | F1-score: 59.46 -> 79.00 | Helped |
| Filter low-agreement caption pairs (ablation) | BRACE benchmark annotations · AudioCaps and Clotho | Fleiss' Kappa: 0.3806 -> 0.9822; 0.4380 -> 0.8422 | Helped |
| Add reference captions to most LALM prompts (ablation) | LALMs · BRACE benchmarks | F1-score | Mixed |
Surprising findings
- Several LALMs (e.g., GAMA) score near or below chance on BRACE-Main, with strong position bias toward caption_0 or caption_1.
- Adding reference captions helps CLAP models but tends to hurt most LALMs.
Limitations
- Benchmark is restricted to existing open-source datasets (AudioCaps, Clotho), limiting diversity across languages, cultures and acoustic environments.
- LALM evaluation is expensive (up to 14 hours per round), so LALM results come from single runs without error bars.
- LALM outputs are distilled to preferences using a secondary LLM, adding a possible source of error.
Experimental setup
Datasets: AudioCaps Clotho
Benchmarks: BRACE-Main BRACE-Hallucination
Metrics: F1-score accuracy by caption pair type
Baselines: MS-CLAP-2022 MS-CLAP-2023 M2D-CLAP LAION-CLAP LTU GAMA Qwen-Audio-Chat Qwen2-Audio-Instruct AF2 GPT-4o-Audio-Preview
Builds on: CLAPScore CLAP FENSE Comp-A AudioCaps Clotho
Closest papers by meaning
- ADIFF: Explaining audio difference using natural language (ICLR 2025 · similarity 0.86)
- Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning (NeurIPS 2025 · similarity 0.86)
- An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching (NeurIPS 2024 · similarity 0.84)
- TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.84)
- OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.84)
- Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.