Research map / Video / Video-language understanding
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoGazeVQA, an egocentric gaze-guided video QA benchmark with MLLM-generated and human-verified intent questions, evaluates gaze-guided prompting strategies on several MLLMs, and shows LoRA fine-tuning and gaze estimation effects.
Problem
Existing video QA benchmarks, including egocentric ones, ignore gaze as a cue for user intent, so MLLMs cannot reliably answer questions about what a camera wearer is looking at and why.
Approach
Builds a gaze-conditioned egocentric VQA benchmark (QA generated by Qwen2.5-VL from frames plus normalized gaze coordinates, then human-verified) and tests three ways of injecting gaze into MLLM prompts: textual coordinates, red-circle visual marks, and sequential gaze salience maps.
Key results
- Qwen2.5-VL-72B with sequential gaze salience map prompting reaches 66.3% average accuracy vs. 60.5% without gaze, while human performance is 83.8%.
- Across the six evaluated MLLMs, gaze-guided prompting improved average accuracy for every model, with gains from +1.3 to +17.8 points (e.g., MiniCPM-o 2.6 from 35.9 to 53.7 with saliency maps).
- LoRA fine-tuning Qwen2.5-VL-7B on about 500 EGTEA gaze-guided QA pairs raised Ego4D+EgoExo average accuracy from 54.0% to 69.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sequential gaze salience map prompting | Qwen2.5-VL-72B · EgoGazeVQA | accuracy (Avg.): 60.5 -> 66.3 | Helped |
| Add sequential gaze salience map prompting | MiniCPM-o 2.6-8B · EgoGazeVQA | accuracy (Avg.): 35.9 -> 53.7 | Helped |
| Add gaze information using the best reported strategy per model | GPT-4o mini, Gemini 2.0 Flash-Lite, LLaVA-NeXT-Video-7B, InternVL2.5-8B, Qwen2.5-VL-7B · EgoGazeVQA | accuracy (Avg.): 57.0 -> 58.8; 51.0 -> 57.2; 54.9 -> 58.3; 58.3 -> 60.6; 49.7 -> 53.4 | Helped |
| Add sequential gaze salience map prompting | GPT-4o mini · EgoGazeVQA, Kitchen | accuracy: ↓0.4 | Hurt |
| Add sequential gaze salience map prompting | InternVL2.5 · EgoGazeVQA, Garage | accuracy: ↓0.9 | Hurt |
| Add sequential gaze salience map prompting | GPT-4o mini · EgoGazeVQA, Mechanics | accuracy: ↓1.9 | Hurt |
| Add sequential gaze salience map prompting | Gemini 2.0 Flash-Lite · EgoGazeVQA, Mechanics | accuracy | No clear effect |
| Crop frames using gaze fixation (GazeMap) | CLIP ViT-H-14 · EgoGazeVQA | accuracy: 21.63 -> 21.79 | No clear effect |
| Crop frames using gaze fixation (GazeMap) | EgoVLP · EgoGazeVQA | accuracy: 22.95 -> 22.65 | Hurt |
| Crop frames using gaze fixation (GazeMap) | BLIP-2 · EgoGazeVQA | accuracy: 27.60 -> 27.21 | Hurt |
| LoRA fine-tune on EGTEA data | Qwen2.5-VL-7B · Ego4D + EgoExo test | accuracy (Average): 54.0 -> 69.5 | Helped |
| LoRA fine-tune on Ego4D + EgoExo data | Qwen2.5-VL-7B · EGTEA test | accuracy (Average): 52.0 -> 65.4 | Helped |
Surprising findings
- Qwen2.5-VL-72B with gaze coordinates from its own prompted estimation (GE) gains only +1.0 average, and gains shrink or reverse when gaze estimation is less accurate (EGTEA GE drops to -0.6).
- Gaze-guided prompting yields the largest gains for MiniCPM-o 2.6, while smaller open-source models benefit less, suggesting model scale matters for using gaze cues.
Limitations
- Most eye-tracked egocentric videos are indoor, limiting generalization to outdoor settings.
- Audio modality not explored; videos are short to medium length, so long-video behavior is not evaluated.
- Heavy camera motion and gaze saccades can mislead models even with gaze cues.
Experimental setup
Datasets: Ego4D EgoExo4D EGTEA Gaze+
Benchmarks: EgoGazeVQA
Metrics: accuracy mean squared error (gaze estimation)
Baselines: GPT-4o mini Gemini 2.0 Flash-Lite LLaVA-NeXT-Video-7B MiniCPM-o 2.6-8B InternVL2.5-8B Qwen2.5-VL-7B Qwen2.5-VL-72B CLIP ViT-H-14 EgoVLP BLIP-2
Builds on: Ego4D EgoExo4D EGTEA Gaze+ LLaMA-Factory
Closest papers by meaning
- Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding (NeurIPS 2025 · similarity 0.91)
- StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos (NeurIPS 2026 · similarity 0.91)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.89)
- EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.88)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.88)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.