Research map / Vision-language and multimodal / Multimodal LLMs

AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AVGen-Bench, a 235-prompt task-driven benchmark for text-to-audio-video generation with a hybrid multi-granular evaluation suite (specialist models plus MLLMs), evaluating 12 T2AV systems and revealing strong aesthetics but weak fine-grained semantic control.

Problem

Existing audio-video generation benchmarks evaluate audio and video in isolation or use coarse embedding similarity, failing to measure fine-grained joint audio-visual correctness required by realistic text prompts.

Approach

Task-driven prompt curation across 11 real-world categories, decoupled from metric availability, combined with a multi-granular evaluation suite that chains lightweight specialist models (OCR, face clustering, AMT, ASR, SyncNet, VideoPhy2) with MLLM reasoning (Gemini) for fine-grained checks such as text rendering, pitch accuracy, facial consistency, speech, physics and holistic alignment.

Key results

• Veo 3.1-fast achieves the top Overall score of 67.87, with Sora-2 leading Hi-Phy (78.95) and Holistic (88.89) metrics.
• All evaluated models score below 12/100 on pitch accuracy, indicating near-universal failure on musical pitch control.
• Automated metrics correlate with expert human judgments at Pearson 0.83-0.97 on five of six fine-grained dimensions (Pitch Accuracy 0.5544 due to floor effect).

Surprising findings

• Universal failure on incidental text rendering across all evaluated models despite strong visual aesthetics.
• Pitch accuracy is near-zero for all models, with models generating timbre correctly but random notes.

Limitations

• Pitch accuracy metric shows weaker human correlation (0.5544) and low inter-rater agreement (0.3156), attributed to a floor effect.
• Benchmark is small (235 prompts) and relies heavily on Gemini as judge across multiple metrics.
• Resolution and duration are non-uniform across models (e.g., MOVA at 360p, Sora 2 at 12s), complicating comparison.

Experimental setup

Datasets: AVGen-Bench prompts (235 curated prompts)

Benchmarks: AVGen-Bench

Metrics: Visual quality (Q-Align) Audiobox-Aesthetic PQ AV sync offset (Syncformer) Lip sync error (SyncNet) Scene text accuracy Facial consistency Pitch accuracy Speech intelligibility & coherence Physical plausibility (VideoPhy2, semantic physics score) Holistic semantic alignment

Baselines: Sora 2 Veo 3.1 Kling 2.6 Wan 2.6 Seedance-1.5 Pro LTX-2 LTX-2.3 Ovi Wan2.2+HunyuanVideo-Foley Emu3.5+MOVA

Builds on: VBench Q-Align SyncNet VideoPhy2 PhyT2V

Closest papers by meaning

• T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.89)
• ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.88)
• Tell What You Hear From What You See - Video to Audio Generation Through Text (NeurIPS 2024 · similarity 0.88)
• VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.88)
• TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.88)