Research map / Vision-language and multimodal / Human and embodied modeling
EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces audio-synchronized gaze trajectory prediction with a 20k-point dataset and EyEar, a physics-informed dynamical model with a probability density score and loss.
Problem
Predicting human gaze trajectories over an image while synchronized audio narration is heard, a setting not covered by free-viewing, visual search, or VQA gaze tasks, with high inter-individual variability that hurts MSE optimization and point-based metrics.
Approach
Models gaze as a dynamical system whose motion vector is the sum of three forces (inherent motion tendency, vision salient attraction from DeepGaze IIE, and audio semantic attraction from a BriVL-based multimodal attention), and trains/evaluates with a Gaussian kernel density based probability density score and loss instead of point distances.
Key results
- EyEar achieves PDS 0.6138 vs 0.5325 for the best baseline (Chen et al VQA), a 15.3% relative gain.
- EyEar reaches ED 221.6 and DTW 201.3, beating the best baselines (MITR 239.3, Chen et al VQA 228.6).
- Removing the PD loss causes the largest ablation drop (PDS 0.614 to 0.515).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EyEar instead of MITR | gaze trajectory prediction on EyEar-20k · EyEar-20k test set | ED: 239.3 -> 221.6 | Helped |
| Use EyEar instead of Chen et al (VQA) | gaze trajectory prediction on EyEar-20k · EyEar-20k test set | DTW: 228.6 -> 201.3 | Helped |
| Use EyEar instead of G-DINO | gaze trajectory prediction on EyEar-20k · EyEar-20k test set | SMatch: 0.443 -> 0.464 | Helped |
| Use EyEar instead of Chen et al (VQA) | gaze trajectory prediction on EyEar-20k · EyEar-20k test set | PDS: 0.5325 -> 0.6138 | Helped |
| Add vision salience attraction point (ablation) | EyEar · EyEar-20k test set | PDS: 0.602 -> 0.614 | Helped |
| Use DynS instead of feedforward networks (ablation) | EyEar · EyEar-20k test set | PDS: 0.568 -> 0.614 | Helped |
| Add GRU for audio transcript encoding (ablation) | EyEar · EyEar-20k test set | PDS: 0.555 -> 0.614 | Helped |
| Add PD loss (ablation) | EyEar · EyEar-20k test set | PDS: 0.515 -> 0.614 | Helped |
Surprising findings
- Point-based metrics (ED, DTW, ScanMatch) let some models match or exceed Human results, while PDS shows a clear gap to human (0.7243).
- Removing the vision salience term had the smallest effect, suggesting audio drives gaze more than image saliency in this setting.
Limitations
- Gap to human performance remains, especially in large-range saccade motion.
- Only static images with narrative Chinese audio; open audio stimuli and videos not studied.
- Dataset is small (8 subjects) and all data are in Chinese.
Experimental setup
Datasets: EyEar-20k (collected) OSIE COCO-FreeView COCO-Search18 AiR
Benchmarks: EyEar-20k test set
Metrics: ED DTW ScanMatch PDS
Baselines: CLIP BriVL Grounding DINO MITR Chen et al (VS) Gazeformer Chen et al (VQA)
Builds on: Grounding DINO BriVL DeepGaze IIE GRU Transformer
Closest papers by meaning
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.90)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.89)
- StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos (NeurIPS 2026 · similarity 0.88)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.87)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.87)
- DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.