Research map / Video / Action recognition
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
Yin Li, Miao Liu, James M. Rehg
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| joint RGB and flow training instead of RGB only (ablation) | I3D · EGTEA Gaze+ action recognition | video accuracy: 47.26 -> 49.79 | Improved |
| add RGB stream to flow stream (ablation) | I3D · EGTEA Gaze+ action recognition | video accuracy: 38.31 -> 47.26 | Improved |
| joint training instead of late fusion (ablation) | I3D · EGTEA Gaze+ action recognition | video accuracy: 48.84 -> 49.79 | Improved |
| probabilistic gaze modeling instead of deterministic MLE (ablation) | Gaze-action model · EGTEA Gaze+ action recognition | video accuracy: 51.12 -> 53.30 | Improved |
| probabilistic gaze modeling instead of deterministic MLE (ablation) | Gaze-action model · EGTEA Gaze+ gaze estimation | F1: 24.68 -> 32.97 | Improved |
| add dropout after attention-pooled features (ablation) | Probabilistic model · EGTEA Gaze+ action recognition | video accuracy: 53.30 -> 52.12 | Hurt |
| add dropout after attention-pooled features (ablation) | Probabilistic model · EGTEA Gaze+ gaze estimation | F1: 32.97 -> 32.66 | Hurt |
| gaze supervision instead of unsupervised soft attention (ablation) | Attention model · EGTEA Gaze+ action recognition | video accuracy: 50.30 -> 53.30 | Improved |
| soft attention without gaze supervision added to I3D (ablation) | I3D Joint · EGTEA Gaze+ action recognition | video accuracy: 49.79 -> 50.30 | Improved |
| gaze supervision for attention (ablation) | Attention model · EGTEA Gaze+ gaze estimation | F1: 10.27 -> 32.97 | Improved |
| use human gaze at test time instead of predicted gaze | I3D+Gaze vs ours · EGTEA Gaze+ action recognition | video accuracy: 51.21 -> 53.30 | Hurt |
| joint gaze and action modeling instead of gaze-only models | Deep Gaze · EGTEA Gaze+ gaze estimation | F1: 33.51 -> 32.97 | Hurt |
Closest papers by meaning
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.89)
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.88)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.87)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.87)
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.87)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
