Research map / Video / Video segmentation and tracking

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

NeurIPS 2026 · Video › Video segmentation and tracking · approach: Physics-inspired motion prediction · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GazeFlow, a conditional flow matching model that generates whole egocentric gaze trajectories conditioned on V-JEPA 2 visual tokens and learned task tokens, achieving state-of-the-art per-frame accuracy on EGTEA Gaze+ and Ego4D.

Problem

Egocentric gaze prediction is hard because human gaze is stochastic; prior methods model per-frame marginals, autoregressive chains, or regress a single point, missing joint temporal structure and multimodality.

Approach

Model the full gaze trajectory as a joint distribution conditioned on the video via conditional flow matching, with a velocity network that combines frame-local bottom-up visual cross-attention, global top-down task-token cross-attention, and time-aware self-attention with RoPE.

Key results

• On EGTEA Gaze+, GazeFlow reaches AUC 0.964 and F1 0.491 versus 0.956 and 0.421 for the strongest baseline AT, with F1 improved by 16.6%.
• On best-of-K ADE, GazeFlow reduces error from 0.96 (EgoM2P LoRA) to 0.60 on EGTEA Gaze+.
• On Ego4D, GazeFlow achieves AUC 0.958 and F1 0.392 versus 0.954 and 0.355 for AT/GLC-level baselines, improving F1 by 10.4%.

Surprising findings

• Replacing flow matching with a diffusion objective reduces F1 from 0.506 to 0.359 at 5 sampling steps, while flow matching holds up.
• Linear probing shows V-JEPA 2 features at native resolution match Qwen3-VL-8B features in AUC, and the 4:3 input degrades performance.

Limitations

• Failures concentrate on saccades whose endpoint lies outside the camera field of view.
• Conditioning on the full video makes it unsuitable for real-time gaze prediction; the causal variant loses some accuracy.
• Models residual eye movement given recorded head movement; head and eye movements are not jointly modeled.

Experimental setup

Datasets: EGTEA Gaze+ Ego4D (Aria gaze subset)

Benchmarks: EGTEA Gaze+ Ego4D

Metrics: AUC F1 precision recall AAE ADE DTW per-frame displacement ratio JSD

Baselines: GLC AT EgoM2P (zero-shot) EgoM2P (LoRA) Center Bias Random Walk

Builds on: flow matching V-JEPA 2 GLC Attention Transition EgoM2P

Closest papers by meaning

• ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.90)
• Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding (NeurIPS 2025 · similarity 0.89)
• EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.89)
• Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span (NeurIPS 2025 · similarity 0.88)
• DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images (NeurIPS 2025 · similarity 0.87)
• Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.87)