Research map / Video / Video-language understanding
StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StreamGaze, a gaze-guided streaming video QA benchmark (past, present, proactive tasks) built via a fixation-extraction and scanpath-based pipeline over egocentric videos, and evaluates MLLMs against humans.
Problem
Streaming video benchmarks do not test whether MLLMs can interpret and use human gaze signals for temporal reasoning, intention inference, and proactive prediction in egocentric streaming settings.
Approach
A semi-automatic gaze-video QA pipeline extracts fixations (point-wise stability plus Hue-Saturation scene consistency), splits frames into FOV and out-of-FOV regions, uses region-specific visual prompting with InternVL3.5-38B to extract objects, builds scanpaths, and generates past/present/proactive QA with an LLM plus human verification.
Key results
- Best closed-source model GPT-4o reaches 0.535 overall accuracy versus 0.827 for human performance.
- AssistGaze, the gaze-based fine-tuned model, achieves only 0.223 overall accuracy.
- Salience-map gaze prompting for Qwen2.5-VL gives a small average gain (0.454 vs 0.446 without gaze), and no gaze prompting consistently beats the gaze-free baseline.
Surprising findings
- Gaze prompting strategies (textual, visual, salience map) do not consistently outperform the gaze-free baseline for Qwen2.5-VL.
- Adding text chain-of-thought reduces GPT-4o's NFI accuracy from 0.52 to 0.10.
Limitations
- Human verification inter-annotator agreement is moderate (Fleiss' Kappa 0.60), and about 83% of generated objects are judged correct.
- Streaming tasks are converted to offline inference, so true streaming behavior is not evaluated for non-streaming models.
- Pipeline relies on proprietary or large MLLMs (InternVL3.5-38B, Qwen3-VL-30B) and fixed FOV thresholds.
Experimental setup
Datasets: EGTEA+ EgoExoLearn HoloAssist Ego4D-Gaze
Benchmarks: StreamGaze
Metrics: accuracy precision recall
Baselines: GPT-4o Claude Sonnet4 Claude Opus4 AssistGaze Qwen2.5-VL InternVL3.5 MiniCPM-V Kangaroo EgoGPT VITA 1.5 Flash-VStream VideoLLM-online
Builds on: OVO-Bench StreamingBench ViSpeak AssistGaze
Closest papers by meaning
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.91)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.88)
- Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models (CVPR 2026 · similarity 0.88)
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.88)
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.88)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.