Research map / Video / Video-language understanding
SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SVBench, a benchmark of 49,979 temporal multi-turn QA pairs over 1,353 streaming videos for evaluating LVLMs on long-context streaming video understanding, and a fine-tuned InternVL2-based StreamingChat baseline.
Problem
Existing video QA benchmarks use isolated single-turn questions on short clips and do not measure whether LVLMs can sustain temporal, multi-turn reasoning over long streaming videos.
Approach
Builds QA chains per video clip and temporal linkages between consecutive chains, so that questions form temporal multi-turn dialogue paths; evaluates in dialogue mode (full history) and streaming mode (80% probability of jumping to linked questions), scored by LLM-based dimensions and METEOR.
Key results
- GPT-4o achieves the highest overall score on SVBench: 66.29 in dialogue evaluation and 58.17 in streaming evaluation.
- StreamingChat (fine-tuned InternVL2) reaches 59.41 dialogue OS and 53.90 streaming OS, a 28.79% and 26.20% relative improvement over InternVL2.
- Most open-source LVLMs score below 60 on the nine skill categories, and human performance on the dialogue setting is 83.93 OS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune InternVL2 with SVBench temporal dialogue training data | InternVL2 / StreamingChat · SVBench dialogue evaluation | OS: +28.79% | Helped |
| Fine-tune InternVL2 with SVBench temporal dialogue training data | InternVL2 / StreamingChat · SVBench streaming evaluation | OS: +26.20% | Helped |
| Include contextual information from previous QA pairs instead of single-instance QA (ablation) | LVLMs evaluated on SVBench · SVBench dialogue evaluation | METEOR | Helped |
| Include contextual information from previous QA pairs instead of single-instance QA (ablation) | LVLMs evaluated on SVBench · SVBench dialogue evaluation | GPT4-Score | Helped |
Surprising findings
- Streaming evaluation scores are consistently lower than dialogue evaluation scores across all models.
- InternVL2-Llama3-76B judge shows excessive leniency, with more than half of models scoring above 60, while METEOR lacks discrimination.
Limitations
- Annotation pipeline relies on GPT-4o and GPT-4 for generation and quality checks, creating a circular dependency concern partly mitigated by manual annotation.
- Some evaluated models (e.g., Oryx, Long-LLaVA, LongVILA) were not yet included in the main results table at submission.
- Human manual modification is required for temporal linkages, which limits scalability.
Experimental setup
Datasets: YT-Temporal-1B YouCook2 ActivityNet MovieChat Panda-70M Ego4D MMBench MMBench-Video VideoMME MVBench
Benchmarks: SVBench MMBench MMBench-Video VideoMME MVBench
Metrics: METEOR GPT4-Score BLEU-4 ROUGE-L CIDEr Semantic Accuracy Contextual Coherence Logical Consistency Temporal Understanding Informational Completeness
Baselines: MovieChat Video-ChatGPT Video-LLaVA ShareGPT4Video VideoLLaMA2 TimeChat InternVL2 VILA InternLM-XComposer2.5 MiniCPM-V 2.6 Gemini 1.5 Pro GPT-4V
Builds on: InternVL2 LLaVA-1.5 LoRA
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.93)
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.93)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.92)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.91)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.91)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.