Research map / Vision-language and multimodal / Multimodal LLMs
RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RTV-Bench, a 552-video, 4,608-QA benchmark with multi-timestamp questions and hierarchical perception/understanding/reasoning evaluation, showing real-time streaming MLLMs beat offline ones but trail GPT-4o.
Problem
Existing video benchmarks do not adequately evaluate MLLMs' continuous perception, understanding, and reasoning over dynamic, real-time video streams, where answers change as scenes evolve.
Approach
Multi-timestamp QA that repeats the same conceptual question at multiple times with time-varying correct answers, combined with hierarchical basic-to-advanced question sets and a group-aware Score that rewards advanced answers only when prerequisites are correct.
Key results
- IXC2.5-OL (7B, online) reaches 47.33% overall accuracy and 15.40 score, above offline VideoLLaMA2 (39.55% / 7.90) but below GPT-4o (50.02% / 22.10).
- Most evaluated models achieve below 50% accuracy on RTV-Bench.
- Qwen2.5-VL 72B reaches up to 40.78% accuracy at 64 frames; scaling from 3B to 72B yields only about 2-3 points gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an online streaming video model rather than an offline video model | MLLMs evaluated on RTV-Bench · RTV-Bench | Overall accuracy: 39.55% -> 47.33% | Helped |
| Use an online streaming video model rather than an offline video model | MLLMs evaluated on RTV-Bench · RTV-Bench | Overall score: 7.90 -> 15.40 | Helped |
| Use IXC2.5-OL rather than VITA-1.5 | Open-source online video models · RTV-Bench | Overall accuracy: +2.82% | Helped |
| Use IXC2.5-OL rather than VITA-1.5 | Open-source online video models · RTV-Bench | Overall score: +3.6 points | Helped |
| Increase Qwen2.5-VL model scale from 3B to 72B (ablation) | Qwen2.5-VL · RTV-Bench | Overall accuracy: ~2–3 points | Helped |
| Increase sampled input frames from 8 to 64 (ablation) | MLLMs across model scales · RTV-Bench | Overall accuracy | Mixed |
| Increase sampled input frames (ablation) | IXC2.5-OL · RTV-Bench | Overall score | Hurt |
| Increase sampled input frames from 8 to 64 (ablation) | Qwen2.5-VL 72B · RTV-Bench | Overall accuracy: 40.78% with 64 frames | Helped |
Surprising findings
- Increasing the number of input frames does not consistently improve performance and can reduce score, e.g., IXC2.5-OL drops with more frames.
- Model scale gains are moderate, with diminishing returns from 3B to 72B.
Limitations
- Evaluation is limited to the visual modality; audio is not included.
- Improvements to architecture or frame sampling strategies are not explored beyond uniform sampling.
- Results for some models depend on best-of-frame-count reporting, which may not reflect a fixed protocol.
Experimental setup
Datasets: EgoSchema
Benchmarks: RTV-Bench
Metrics: accuracy score (group-aware Q2 metric)
Baselines: GPT-4o Gemini 2.0 Flash Qwen2.5-VL VideoLLaMA2 VideoLLaMA3 LLaVA-OneVision LLaVA-Video VITA-1.5 IXC2.5-OL VideoChat-Online
Builds on: OVO-Bench StreamingBench VStream
Closest papers by meaning
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.91)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.91)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.90)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.