Research map / Video / Video-language understanding
Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dispider, a disentangled video LLM framework for active real-time streaming interaction that separates a lightweight proactive decision module (Qwen2-1.5B) from an asynchronous precise response module (Qwen2-7B).
Problem
Real-time streaming video LLMs must decide when to respond, track continuous video, and generate answers, but a single autoregressive LLM cannot perceive and react in parallel, causing latency and blocked perception during responses.
Approach
Disentangle perception/decision from reaction: a compact LLM with scene-based clip segmentation and a <TODO>/<ANS> token scheme decides when to respond and retrieves relevant clips, while a larger LLM generates answers asynchronously without interrupting video processing.
Key results
- Achieves 55.6 on EgoSchema full set, 61.7 on MLVU, and 57.2 on VideoMME, competitive with larger offline video LLMs.
- On ETBench streaming setting, temporal grounding F1 reaches 36.1 versus 13.2 for VideoLLM-online.
- Scene-based segmentation improves MLVU from 59.8 to 61.7 and VideoMME from 55.4 to 57.2 over uniform 16-frame clips.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace uniform 16-frame clip segmentation with scene-based segmentation (ablation) | Dispider · MLVU | accuracy: 59.8 -> 61.7 | Helped |
| replace uniform 16-frame clip segmentation with scene-based segmentation (ablation) | Dispider · VideoMME | accuracy: 55.4 -> 57.2 | Helped |
| replace uniform 16-frame clip segmentation with scene-based segmentation (ablation) | Dispider · ETBench streaming TVG | F1: 34.5 -> 36.1 | Helped |
| replace uniform 16-frame clip segmentation with scene-based segmentation (ablation) | Dispider · ETBench streaming DVC | F1: 33.1 -> 33.8 | Helped |
| replace uniform 16-frame clip segmentation with scene-based segmentation (ablation) | Dispider · ETBench streaming DVC | sentence similarity: 18.1 -> 18.9 | Helped |
| add the ⟨ANS⟩, ⟨TODO⟩, and ⟨SILENT⟩ tokens (ablation) | Dispider · ETBench streaming | TVG F1: 20.1 -> 36.1 | Helped |
| add the ⟨ANS⟩, ⟨TODO⟩, and ⟨SILENT⟩ tokens (ablation) | Dispider · ETBench streaming | DVC F1: 19.7 -> 33.8 | Helped |
| add the ⟨ANS⟩, ⟨TODO⟩, and ⟨SILENT⟩ tokens (ablation) | Dispider · ETBench streaming | DVC sentence similarity: 12.3 -> 18.9 | Helped |
Surprising findings
- Streaming-mode dense captioning and step localization scores exceed the conventional-setting scores on ETBench, suggesting streaming interaction may help understanding.
Limitations
- Efficiency claims are not backed by quantitative latency or throughput measurements in the text.
- Streaming evaluation is limited to a subset of ETBench subtasks and relies on a single prior baseline (VideoLLM-online) for streaming comparison.
Experimental setup
Datasets: GroundVQA ET-Instruct VideoChatGPT LLaVA-Next-Video ETBench EgoSchema VideoMME MLVU
Benchmarks: EgoSchema MLVU VideoMME ETBench
Metrics: accuracy F1 sentence similarity
Baselines: VideoLLM-online VideoXL Kangaroo Video-CCAM LongVA ETChat TimeChat MovieChat
Builds on: VideoLLM-online SigLIP Flash-VStream token compression
Closest papers by meaning
- OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts (CVPR 2025 · similarity 0.85)
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant (NeurIPS 2025 · similarity 0.85)
- StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos (NeurIPS 2026 · similarity 0.85)
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.84)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.84)
- OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.