Research map / Vision-language and multimodal / Human and embodied modeling
Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-only gaze regularizer that aligns VLM attention with aggregated, occlusion-filtered gaze heatmaps via KL divergence, improving future event prediction and current activity understanding on Ego4D across five VLM architectures.
Problem
Egocentric VLMs generating fine-grained descriptions of current activities and future events lack human-like attention focus; gaze is available in training data but unavailable at deployment.
Approach
Use human gaze only at training time: convert gaze points into temporally aggregated, optical-flow occlusion-filtered heatmaps, pool them to patch-level distributions, and add a KL term aligning a global-query attention distribution with them alongside the cross-entropy captioning loss.
Key results
- Gaze regularization improves future prediction semantic similarity by 8.9% to 10.5% absolute across five VLM architectures (e.g., OpenFlamingo 0.6525 to 0.7505).
- Current activity understanding improves by 4.9% to 6.9% (e.g., OpenFlamingo 0.7176 to 0.7848).
- Human-evaluated hallucination rate C_I drops from 0.205 to 0.140 on 200 samples; top-10 attention-gaze overlap rises from 42% to 68%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gaze-regularized attention during training | OpenFlamingo · Ego4D future prediction (τa=2s) | semantic similarity: 0.6525 -> 0.7505 | Helped |
| add gaze-regularized attention during training | OpenFlamingo · Ego4D activity understanding | semantic similarity: 0.7176 -> 0.7848 | Helped |
| add gaze-regularized attention during training | VLM backbones · Ego4D future prediction and activity understanding | semantic similarity: +8.9% to 10.5%; +4.9% to 6.9% | Helped |
| add gaze-regularized attention during training | OpenFlamingo · EGTEA+ Gaze future prediction | semantic similarity: 0.6501 -> 0.7305 | Helped |
| add gaze-regularized attention during training | OpenFlamingo · EGTEA+ Gaze activity understanding | semantic similarity: 0.6963 -> 0.7527 | Helped |
| remove gaze regularization (λ=0) (ablation) | OpenFlamingo · Ego4D future prediction (τa=2s) | semantic similarity: 0.7505 -> 0.6317 | Hurt |
| remove gaze regularization (λ=0) (ablation) | OpenFlamingo · Ego4D activity understanding | semantic similarity: 0.7848 -> 0.6917 | Hurt |
| increase gaze regularization strength from λ=100 to λ=1000 (ablation) | OpenFlamingo · Ego4D future prediction (τa=2s) | semantic similarity: 0.7505 -> 0.7354 | Hurt |
| increase gaze regularization strength from λ=100 to λ=1000 (ablation) | OpenFlamingo · Ego4D activity understanding | semantic similarity: 0.7848 -> 0.7721 | Hurt |
| extend anticipation horizon from 2s to 5s (ablation) | OpenFlamingo · Ego4D future prediction | semantic similarity: 0.7505 -> 0.7315 | Hurt |
| remove occlusion filtering from gaze aggregation (ablation) | Gaze-Regularized OpenFlamingo · Ego4D future prediction | semantic similarity: 0.7505 -> 0.7298 | Hurt |
| add gaze-regularized attention during training | OpenFlamingo · Ego4D hallucination evaluation | C_I: 0.205 -> 0.140 | Helped |
Surprising findings
- Gaze regularization with lambda=0 (no gaze) already yields most of the gain relative to lambda=100 in some reports is not claimed; instead, excessive lambda=1000 slightly reduces performance versus lambda=100.
- The gains persist on the out-of-distribution EGTEA+ Gaze dataset without fine-tuning, with 7-9% future prediction gains.
Limitations
- Ground-truth captions are generated by GPT-4V with only 74% annotation-gaze alignment under strict matching and a small human-verified subset.
- Gains depend on the regularization weight lambda, and excessive regularization can constrain attention to task-relevant regions not captured by gaze.
- Hallucination evaluation relies on a small human study of 200 examples.
Experimental setup
Datasets: Ego4D (gaze subset) EGTEA+ Gaze
Benchmarks: Ego4D gaze subset (held-out) EGTEA+ Gaze (out-of-distribution)
Metrics: Sentence-BERT semantic similarity C_I hallucination rate (human-evaluated) top-10 attention-gaze overlap
Baselines: OpenFlamingo base (no gaze regularization) Modified OpenFlamingo LaViLa Narrator InternVL OpenLLaVA
Builds on: Flamingo LaViLa InternVL LLaVA-NeXT RAFT
Closest papers by meaning
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.91)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.91)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.89)
- Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span (NeurIPS 2025 · similarity 0.88)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.86)
- Voila-A: Aligning Vision-Language Models with User's Gaze Attention (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.