Research map / Vision-language and multimodal / Multimodal LLMs

TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TTA-Bench, a multi-dimensional benchmark (accuracy, efficiency, generalization, robustness, fairness, bias, toxicity) with 2,999 prompts and over 118k human annotations for evaluating ten text-to-audio models.

Problem

Existing text-to-audio evaluation focuses mainly on perceptual quality and misses robustness, generalization, fairness, bias and toxicity, and relies on narrow datasets and inconsistent subjective protocols.

Approach

A holistic benchmark spanning functional quality, reliability and social responsibility, combining objective metrics (AES, CLAP, RTF, robustness/fairness ratios, MAD) with expert and crowd human ratings, including reference-free evaluation and audio-specific toxicity categories adapted from I2P.

Key results

• Tango 2 achieves the best overall accuracy and generalization performance across crowd and expert MOS, with CLAP 0.46 on the accuracy set.
• AudioLDM has the lowest overall toxicity rate (0.697), while Tango 2 has the highest (0.950).
• Make-An-Audio 2 and Stable Audio Open are most sensitive to perturbations (robustness ratios around 0.85-0.9), while AudioGen and Tango 2 are closest to 1.

Surprising findings

• Tango 2, the top performer on quality and generalization, shows the highest toxicity rate across most categories.
• Models generally score highest on original dataset captions and degrade under template or manual prompts, but Tango and Make-An-Audio 2 stay stable.

Limitations

• Human annotation relies on 13 raters (3 expert, 10 general), a small rater pool.
• Toxicity and bias evaluation use limited prompt sets (e.g., 300 bias prompts, 300 toxicity prompts) and gender detection via a commercial API.
• Evaluated models are mostly open-source models from 2023-2025, with evaluation limited to ten systems.

Experimental setup

Datasets: AudioCaps AudioSet I2P NL-Augmenter

Benchmarks: TTA-Bench

Metrics: AES CLAP score real-time factor robustness score fairness score MAD toxicity rate MOS (crowd and expert)

Baselines: AudioGen AudioLDM AudioLDM 2 Auffusion MAGNeT Make-An-Audio Make-An-Audio 2 Stable Audio Open Tango Tango 2

Builds on: AudioCaps AudioSet I2P Audiobox-Aesthetic CLAP NL-Augmenter

Closest papers by meaning

• ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.90)
• AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.88)
• EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.87)
• Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.87)
• PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios (ICML 2026 · similarity 0.86)
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.86)