Research map / Video / Video-language understanding
Online Video Understanding: OVBench and VideoChat-Online
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OVBench, a 16-subtask QA benchmark for online spatiotemporal video understanding, and VideoChat-Online, a 4B MLLM with a Pyramid Memory Bank and offline-to-online training that outperforms larger offline and online models.
Problem
Existing video understanding benchmarks and MLLMs are built for offline video, but real-world online streaming scenarios require real-time, time-dependent spatiotemporal understanding of continuous video streams with efficient memory.
Approach
A pyramid memory bank that stores frames at decreasing resolution and increasing sampling rate across layers, aligned with KV cache for eviction, combined with an offline-to-online curriculum that trains on interleaved dialogue-format online spatiotemporal data.
Key results
- VideoChat-Online (4B) reaches 54.9% overall on OVBench in streaming setting, versus 31.2% for Flash-VStream (7B), a 23.7-point gain.
- VideoChat-Online (4B) reaches 53.9% on OVBench sliding-window setting, 4.19% above Qwen2-VL-7B (49.7%).
- On offline benchmarks, VideoChat-Online scores 54.7 on EgoSchema, 60.8 on MLVU, 54.4 on VideoMME and 65.2 on MVBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoChat-Online instead of the strongest reported baseline | VideoChat-Online · OVBench, streaming | average: 31.2 -> 54.9 | Helped |
| Add the high-frequency memory queue mt (ablation) | VideoChat-Online · OVBench | average: 54.2 -> 54.9 | Helped |
| Add the main memory queue mmain (ablation) | VideoChat-Online · OVBench | average: 54.3 -> 54.9 | Helped |
| Add the spatial-detail memory queue ms (ablation) | VideoChat-Online · OVBench | average: 53.7 -> 54.9 | Helped |
| Replace Token Merge with the proposed memory update policy (ablation) | VideoChat-Online · OVBench | average: 51.5 -> 54.9 | Helped |
| Replace First In First Out with the proposed memory update policy (ablation) | VideoChat-Online · OVBench | average: 54.0 -> 54.9 | Helped |
| Replace Uniform Sample with the proposed memory update policy (ablation) | VideoChat-Online · OVBench | average: 52.1 -> 54.9 | Helped |
| Add memory compression rather than inputting raw video data (ablation) | VideoChat-Online · OVBench | average: 49.3 -> 54.9 | Helped |
| Add online training to offline training (ablation) | VideoChat-Online · OVBench | average: 44.12 -> 52.42 | Helped |
| Add interleaved data format to offline training (ablation) | VideoChat-Online · OVBench | average: 44.12 -> 51.84 | Helped |
| Use progressive training with interleaved data instead of joint training with interleaved data (ablation) | VideoChat-Online · OVBench | average: 51.84 -> 53.89 | Helped |
Surprising findings
- VideoLLM-Online performs near zero on several OVBench subtasks because it fails to follow instructions accurately, generating empty or redundant output.
- Offline MLLMs in a sliding-window setting outperform native online streaming models on OVBench, indicating offline-to-online knowledge transfer.
Limitations
- Online performance still lags behind the best offline MLLMs on several OVBench subtasks such as spatial and object-position tasks.
- Streaming evaluation degrades for models as irrelevant context accumulates, and the memory design is evaluated only on this model.
Experimental setup
Datasets: TimeChat-IT DiDeMo QuerYD HiREST Charades-STA LaSOT GOT10K COIN ActivityNet Captions ViTT YouCook2 AVA
Benchmarks: OVBench EgoSchema MLVU VideoMME MVBench LongVideoBench
Metrics: accuracy
Baselines: Gemini-1.5-Flash InternVL2 LLaVA-OneVision Qwen2-VL LongVA MiniCPM-V2.6 VideoLLM-Online MovieChat Flash-VStream
Builds on: InternVL2 VideoChat2 MovieChat VideoLLM-Online Flash-VStream
Closest papers by meaning
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.90)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.89)
- VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · similarity 0.89)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.