Research map / Video / Video-language understanding
Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ESTP-Bench with the ESTP-F1 metric for proactive, just-in-time question answering over ego-streaming video, plus VideoLLM-EyeWO, a LoRA-trained streaming Video-LLM with a multi-stage training strategy and proactive dynamic compression.
Problem
Existing streaming and online video LLMs cannot jointly handle proactive answering to diverse, evolving questions with well-timed responses while staying synchronized with incoming frames and remaining memory-efficient.
Approach
Combines a three-stage synthetic data engine (one-to-one, one-to-many via RAG, many-to-many multi-turn), a staged training curriculum (passive interval responsiveness, ask-for-high-res-frame with just-in-time answering, multi-turn coherence), and a proactive dynamic compression scheme that inserts compression tokens after responses and high/low-resolution segments to shrink the KV cache.
Key results
- On ESTP-Bench, VideoLLM-EyeWO reaches an overall ESTP-F1 of 34.7, versus 18.1 for the best offline baseline (MiniCPM-V, polling) and 17.8 for MMDuet.
- With proactive dynamic compression, KV cache size on contextual questions falls from 31199.5 (LIVE) to 3242.8, about 0.1% of the baseline.
- On OVO-Bench, the model averages 32.76 on real-time perception versus 20.79 for VideoLLM-online.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoLLM-EyeWO instead of MiniCPM-V with Polling Strategy | ESTP-Bench · ESTP-Bench overall | ESTP-F1: +11.8% | Helped |
| Use VideoLLM-EyeWO instead of VideoLLM-online | OVO-Bench · OVO-Bench real-time perception | Average: 20.79 -> 32.76 | Helped |
| Use VideoLLM-EyeWO instead of VideoLLM-online | OVO-Bench · OVO-Bench backward tracing | Average: 17.73 -> 28.00 | Helped |
| Use VideoLLM-EyeWO instead of VideoLLM-online | QAEGO4D-MC · QAEGO4D-MC online | Score: 29.80 -> 36.20 | Helped |
| Use VideoLLM-EyeWO instead of VideoLLM-online | QAEGO4D-MC · QAEGO4D-MC offline | Score: 30.20 -> 33.00 | Helped |
| Add ESTP-IT training data (ablation) | LIVE · ESTP-Bench single question | Performance: 14.9 -> 22.0 | Helped |
| Add ESTP-IT training data (ablation) | LIVE · ESTP-Bench contextual question | Performance: 18.9 -> 25.7 | Helped |
| Add Stage-0 interval-responsiveness training (ablation) | LIVE + ESTP-IT · ESTP-Bench single question | Performance: 22.0 -> 24.9 | Helped |
| Add Stage-0 interval-responsiveness training (ablation) | LIVE + ESTP-IT · ESTP-Bench contextual question | Performance: 25.7 -> 23.0 | Hurt |
| Add high-resolution frame requests and initial compression (ablation) | Stage-0 · ESTP-Bench single question | Performance: 24.9 -> 34.0 | Helped |
| Add high-resolution frame requests and initial compression (ablation) | Stage-0 · ESTP-Bench contextual question | Performance: 23.0 -> 38.7 | Helped |
| Add multi-turn coherence training and refined compression (ablation) | Stage-0 + Stage-1 · ESTP-Bench single question | Performance: 34.0 -> 33.2 | Hurt |
Surprising findings
- Offline MLLMs with a polling strategy generally outperform the Response-in-Last setting, and some models (e.g., InternVL-V2) score very low under polling.
- Recall and precision are prevalently negatively correlated across models, with MMDuet achieving high recall but low precision.
Limitations
- Benchmark questions are drawn from a limited set of Ego4D validation videos (890) and the LLM judge is unspecified.
- The approach is built on the VideoLLM-online architecture and trained with LoRA, so generality to other architectures is untested.
Experimental setup
Datasets: Ego4D Ego4D GoalStep
Benchmarks: ESTP-Bench OVO-Bench QAEGO4D
Metrics: ESTP-F1 accuracy recall precision KV cache size actions per second
Baselines: LLaVA-OneVision Qwen2-VL MiniCPM-V LLaVA-NeXT-Video InternVL-V2 CLIP LaViLa EgoVLP VideoLLM-online MMDuet
Builds on: VideoLLM-online LIVE MMDuet
Closest papers by meaning
- StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos (NeurIPS 2026 · similarity 0.86)
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.85)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.85)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.84)
- IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams (NeurIPS 2026 · similarity 0.84)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.