Research map / Video / Video-language understanding
A Benchmark for Omni-Modal Reasoning in Long Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LongShOTBench, a rubric-graded, intent-driven omni-modal long-video QA benchmark (274 videos, ~41 min avg), and LongShOTAgent, a training-free search-refine-verify agent; 105 models evaluated.
Problem
Existing long-video benchmarks lack joint coverage of visual, speech and non-speech audio, open-ended multi-turn intent-driven questions, and interpretable scoring; MCQ and holistic LLM scoring hide failure sources.
Approach
Scenario/intent-driven question generation from cross-modal captions, with weighted criterion-level rubrics verified by binary LLM checks against human-validated gold answers, plus a training-free ReAct-style agent (search_video, refine_video, verify_claim) that uses modality specialists.
Key results
- LongShOTAgent (training-free) reaches 66.64% overall, the highest among 105 evaluated models.
- Best open-source model Qwen3-Omni-30B-A3B-Thinking reaches 64.05%; best closed-source Gemini 3.1 Pro Preview reaches 55.63%.
- 3-verifier ensemble achieves Cohen's kappa 0.75 and Pearson r 0.92 against human consensus on 400 criterion decisions.
Surprising findings
- Models score higher on Reasoning than on Core Perception across top systems (e.g., Qwen3-Omni-Thinking 72.86 RE vs 51.44 CP).
- Vision-centric agents VideoMind, Video-RAG and Vgent score below 15% on the benchmark.
Limitations
- Non-speech audio is the weakest modality even for best systems, indicating open grounding challenges.
- Verifier-based scoring depends on LLM judges, with agreement with humans only moderate (kappa 0.75).
Experimental setup
Datasets: public long-video collections (repurposed) YouTube videos (manually curated) Video-MME WorldSense
Benchmarks: LongShOTBench Video-MME WorldSense
Metrics: rubric-weighted score (0-100) Cohen's kappa Pearson r Spearman rho
Baselines: Gemini 3.1 Pro Preview Qwen3-Omni-30B-A3B-Thinking Qwen3-VL-235B-A22B-Thinking VideoMind Video-RAG Vgent
Builds on: VideoMind Video-RAG Vgent
Closest papers by meaning
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.93)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.93)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.93)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.91)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.91)
- LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.