Research map / Video / Video-language understanding
OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OVO-Bench, a benchmark of 2,814 timestamped QA samples over 644 videos that evaluates Video-LLMs on backward tracing, real-time perception, and forward active responding for online video understanding.
Problem
Existing video benchmarks evaluate offline models with full-video access and do not test temporal awareness, i.e., answering based on the timestamp at which a question is posed, which online video assistants need.
Approach
Defines three online understanding modes (backward tracing, real-time visual perception, forward active responding) with 12 tasks, builds timestamped meta-annotations via semi-automatic generation plus human curation, and evaluates offline models by truncating video to the query time and online models via dense multi-trigger querying.
Key results
- Best proprietary offline model Gemini 1.5 Pro reaches 63.00 overall average versus 92.81 for human agents.
- Hallucination detection (HLD) for Gemini 1.5 Pro is 52.69% vs 91.37% for humans.
- Streaming model Flash-VStream-7B averages 33.61 overall, below its offline counterparts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use offline video-LLMs for online-style real-time perception queries | Video-LLMs · OVO-Bench Real-Time Visual Perception | Avg. | Helped |
| Use streaming online video models instead of offline video-LLMs | Video-LLMs · OVO-Bench | Overall Avg.: 63.00; 33.61 | Hurt |
| Process video frame by frame under strict latency constraints | Flash-VStream-7B · OVO-Bench Real-Time Visual Perception | Avg.: 69.32; 28.37 | Hurt |
| Use offline models for Forward Active Responding rather than immediate-response tasks | Video-LLMs · OVO-Bench Forward Active Responding | Avg.: 57.15 | Hurt |
| Increase the number of sampled video frames | Video-LLMs · OVO-Bench inference latency study | inference latency | Hurt |
| Rely on current Video-LLMs without temporal prioritization | Video-LLMs · OVO-Bench Real-Time Visual Perception | accuracy: 58.43%; 66.97% | Hurt |
Surprising findings
- Blind GPT-4-turbo without video reaches 52.92 on backward-tracing/hallucination-style tasks, showing textual shortcuts.
- Offline models prompted with truncated video clips outperform dedicated online streaming models on real-time perception tasks.
Limitations
- Forward active responding is evaluated via prompted offline models, which the authors note fails to capture ongoing query information.
- Online streaming models collapse on the forward active responding evaluation pipeline.
- Timestamps for some tasks rely on coarse semi-automatic generation from Gemini-1.5 Pro, requiring human refinement.
Experimental setup
Datasets: QA-Ego4D OpenEQA STAR YouCook2 CrossTask HiREST COIN Perception-Test Thumos MovieNet Ego4D
Benchmarks: OVO-Bench
Metrics: accuracy multi-trigger scoring for forward active responding inference latency
Baselines: GPT-4o Gemini-1.5-Pro Qwen2-VL-72B Qwen2-VL-7B LLaVA-Video-7B LLaVA-OneVision-7B InternVL-V2-8B LongVU-7B Flash-VStream-7B VideoLLM-online-8B Dispider GPT-4-turbo
Builds on: Chain-of-Thought STAR Ego4D Video-MME
Closest papers by meaning
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- Online Video Understanding: OVBench and VideoChat-Online (CVPR 2025 · similarity 0.88)
- VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos (CVPR 2025 · similarity 0.87)
- RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video (NeurIPS 2025 · similarity 0.87)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.