Research map / Vision-language and multimodal / Multimodal LLMs

OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniVideoBench, a 1000-QA benchmark over 628 real-world videos (seconds to 30 min) that tests synergistic audio-visual reasoning with step-by-step traces, and finds large gaps to humans and closed-source models.

Problem

Existing video benchmarks either focus on short clips or treat audio as auxiliary, so they fail to evaluate synergistic, logically consistent audio-visual reasoning in omni-modal MLLMs.

Approach

A manually annotated, multiple-choice benchmark where each question requires both audio and visual cues, filtered by MLLMs and an LLM to remove unimodal shortcuts and text-only answerable items, with atomic modality-tagged reasoning chains and carefully controlled distractor design.

Key results

• Gemini-2.5-Pro reaches 58.90% average accuracy on OmniVideoBench, the best among non-Gemini-3 models, while human annotators score 82.69%.
• Gemini-2.5-Pro gets only 38.46% on music-dominated videos versus 57.72% on sound and 61.66% on speech.
• Qwen2.5-Omni-7B drops from 29.3% (audio+visual) to 26.4% with visual-only input; open-ended QA lowers Gemini-2.0-Flash from 41.50% to 27.06% and Qwen2.5-Omni-7B from 29.30% to 17.25%.

Surprising findings

• Qwen2.5-Omni-7B with audio performs worse than Qwen2.5-VL-7B given ASR transcripts, indicating weak audio integration.
• Disabling audio drops Gemini-2.0-Flash to near-random levels, while ASR text helps but cannot replace audio comprehension for music and sound.

Limitations

• Benchmark is multiple-choice, which may overstate performance relative to open-ended QA.
• Relies on Gemini 2.0 Flash and DeepSeek-V3.1 for filtering, which may introduce model-specific biases.
• Music-related subset is small (91 QA pairs), limiting per-category reliability.

Experimental setup

Benchmarks: OmniVideoBench Daily-Omni

Metrics: accuracy

Baselines: Gemini-2.5-Pro Gemini-2.5-Flash Gemini-2.0-Flash Gemini-3.0-Pro Gemini-3.0-Flash Qwen3-Omni-30B-A3B Qwen2.5-Omni-7B OmniVinci-9B Baichuan-Omni-1.5 HumanOmni-7B VITA-1.5-7B MiniCPM-o

Builds on: Daily-Omni WorldSense OmniBench AV-Odyssey

Closest papers by meaning

• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.95)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.94)
• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.93)
• OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.92)
• OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.92)
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)