Research map / Video / Video generation
T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation
Zhe Cao, Tao Wang, Jiaming Wang, Yanghai Wang, Yuanxing Zhang, Jiahao Wang, Jialu Chen, Miao Deng, Chenxi Liao, Yize Zhang, Yubin Guo, Zhaoxiang Zhang and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase prompt event structure from single event to long-narrative (4+ events) (ablation) | Wan-2.6 · T2AV-Compass event occurrence failure rate | failure rate (%): 22% -> 63% | Hurt |
| increase prompt event structure to long-narrative (4+ events) (ablation) | LTX-2 · T2AV-Compass event occurrence failure rate | failure rate (%): reaches 80% failure | Hurt |
| increase visual subject count from single to crowd (ablation) | T2AV models · T2AV-Compass subject presence failure rate | failure rate (%): 8-10% -> 21-44% | Hurt |
| use closed-source end-to-end models rather than open-source or cascaded | T2AV systems · T2AV-Compass subjective Average | Average score: Veo-3.1 70.29 vs LTX-2 63.72 (best open-source) | Improved |
| use cascaded T2V then V2A pipeline instead of end-to-end | Wan-2.2 + Hunyuan-Foley · T2AV-Compass Video Realism vs alignment | Video Realism: 89.63 Video Realism, but lags in audio-visual alignment | Mixed |
| evaluate audio realism versus video realism across models | T2AV systems · T2AV-Compass realism | Audio Realism: best Audio Realism 53.84 vs best Video Realism 89.95 | Hurt |
| use Gemini-2.5-Pro as judge instead of Gemini-2.5-Flash or Qwen3-Omni-Flash (ablation) | MLLM-as-a-Judge · Human agreement, 50 prompts | Overall L1 distance (lower better): 1.087 vs 1.207 and 1.473 | Improved |
| use Gemini-2.5-Pro as judge for audio realism | MLLM-as-a-Judge · Human agreement, Audio Realism | L1 distance: 1.420 vs Inter-Human 1.042 | Hurt |
| repeat judging three times with Gemini 2.5 Pro (ablation) | MLLM judge pipeline · 500 video samples | coefficient of variation: CV <= 1.02% | No clear effect |
| swap judge to Gemini 2.5 Flash for five-system ranking (ablation) | MLLM judge pipeline · five-system ranking | Spearman rho: rho = 0.9248 | No clear effect |
Closest papers by meaning
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.93)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.89)
- TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.89)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
- Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
