Research map / Vision-language and multimodal / Multimodal LLMs

JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces JointAVBench, a 2,853-question multiple-choice benchmark built by an automated caption-QA-QC pipeline (with human verification) to test joint audio-visual reasoning in Omni-LLMs across 15 tasks, 4 audio types, and 3 scene spans.

Problem

Existing audio-visual benchmarks lack strict audio-visual dependency, cover few audio types, and rarely test multi-scene reasoning, so Omni-LLMs' joint audio-visual reasoning is not rigorously evaluated.

Approach

A three-stage semi-automated pipeline (omni-modal captioning per scene, template/LLM QA generation restricted to required modalities and scenes, multi-step LLM quality control plus human verification) that yields questions requiring both audio and video, organized by cognitive dimension, audio type, and scene span.

Key results

• Best Omni-LLM Gemini2.5-Pro reaches 65.3% average accuracy on JointAVBench; Qwen3-Omni (30B) reaches 63.6%.
• Omni-LLMs outperform Video-LLMs (best 51.7% InternVL-2.5) and Audio-LLMs (best 45.6% Kimi-Audio) on average.
• Automatic AV-correlation score for JointAVBench is 93.5% (human 94.5%), versus 56.7% for Music-AVQA and 62.9% for WorldSense.

Surprising findings

• Omni-LLMs fall behind single-modality models on emotion-based tasks and on spatial tasks like SOOG and SOER.
• Qwen2.5-Omni scores 35.3 on SPL with audio-visual input, lower than it does on some single-modality inputs.

Limitations

• Dataset derived solely from SF20K movies, which may bias distribution.
• Automatic pipeline may introduce label or caption errors, and some questions may have partially plausible options despite audits.
• Evaluation limited to selected representative MLLMs due to compute constraints; proprietary models may have seen overlapping content.

Experimental setup

Datasets: Short-Films 20K (SF20K)

Benchmarks: JointAVBench

Metrics: accuracy

Baselines: Qwen2.5-Omni Qwen3-Omni Gemini2.5-Pro Gemini2.5-Flash VideoLLaMA2 video-SALMONN-2+ OneLLM AVicuna Qwen2.5-VL LLaVA-Video VideoLLaMA3 InternVL-2.5

Builds on: Panda-70M Short-Films 20K

Closest papers by meaning

• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.95)
• HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.93)
• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.93)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.92)
• OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.92)
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.91)