Research map / Video / Video segmentation and tracking
Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes unsupervised mistake detection in egocentric videos by comparing gaze trajectories predicted from video and partial gaze with the tracked gaze, using a new gaze completion model with Gaze-Frame Correlation fusion.
Problem
Detecting mistakes in egocentric videos of skilled human activities without mistake labels or temporal action annotations.
Approach
Mistakes are flagged when eye-gaze becomes unpredictable: a gaze completion transformer, conditioned on video and the first half of a gaze trajectory, predicts the remaining gaze, and deviations from the tracked gaze (scored via heatmap likelihood) indicate mistakes; a correlation fusion module attends from gaze tokens to each local visual token.
Key results
- On EPIC-Tent, the unsupervised method reaches AUC 0.69 and F1 0.51, versus 0.61 AUC for GLC in the unsupervised setting.
- On EPIC-Tent one-class setting, the method (AUC 0.69, F1 0.52) matches the best fully supervised C2F AUC of 0.72 with a comparable F1 (0.54 vs 0.58 with late fusion).
- The method is ranked first on the HoloAssist Mistake Detection challenge; relative gains of up to +14%, +11%, and +5% over baselines on EPIC-Tent, HoloAssist and IndustReal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use gaze completion instead of GLC gaze prediction | unsupervised mistake detection · EPIC-Tent | AUC: 0.61 -> 0.69 | Helped |
| use gaze completion instead of GLC gaze prediction | unsupervised mistake detection · HoloAssist | AUC: 0.54 -> 0.59 | Helped |
| use gaze completion instead of GLC gaze prediction | unsupervised mistake detection · IndustReal | AUC: 0.58 -> 0.62 | Helped |
| use gaze completion instead of GLC gaze prediction | unsupervised mistake detection · EPIC-Tent | F1: 0.44 -> 0.51 | Helped |
| use gaze completion instead of GLC gaze prediction | unsupervised mistake detection · HoloAssist | F1: 0.10 -> 0.18 | Helped |
| use gaze completion instead of GLC gaze prediction | unsupervised mistake detection · IndustReal | F1: 0.21 -> 0.21 | No clear effect |
| replace entropy scoring with Euclidean distance scoring (ablation) | gaze completion mistake detection · EPIC-Tent | AUC: 0.51 -> 0.55 | Helped |
| replace entropy scoring with DTW scoring (ablation) | gaze completion mistake detection · EPIC-Tent | AUC: 0.51 -> 0.56 | Helped |
| replace entropy scoring with heatmap scoring (ablation) | gaze completion mistake detection · EPIC-Tent | AUC: 0.51 -> 0.57 | Helped |
| add channel fusion (ablation) | heatmap-scored gaze completion · EPIC-Tent | AUC: 0.57 -> 0.63 | Helped |
| add correlation fusion (ablation) | heatmap-scored gaze completion · EPIC-Tent | AUC: 0.57 -> 0.65 | Helped |
| add channel and correlation fusion (ablation) | heatmap-scored gaze completion · EPIC-Tent | AUC: 0.57 -> 0.69 | Helped |
Surprising findings
- Gaze-based mistake detection is most effective for complex actions, low self-rated confidence, and hand-eye coordination/object manipulation actions, and less effective for trivial tasks.
- The unsupervised method matches the AUC of the best fully supervised approach on EPIC-Tent without using any mistake labels.
Limitations
- Relies on a gaze tracker being available on the wearable device at test time.
- Absolute F1 and precision remain low on HoloAssist and IndustReal (e.g., F1 0.18 unsupervised on HoloAssist).
- Some correlations reported (action difficulty p<0.057) are borderline in significance.
Experimental setup
Datasets: EPIC-Tent HoloAssist IndustReal
Benchmarks: EPIC-Tent HoloAssist IndustReal
Metrics: ROC-AUC F1 precision recall
Baselines: TimeSformer C2F TrajREC MoCoDAD GLC
Builds on: GLC TrajREC MoCoDAD TimeSformer C2F
Closest papers by meaning
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.86)
- Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding (NeurIPS 2025 · similarity 0.86)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.85)
- StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos (NeurIPS 2026 · similarity 0.84)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.84)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.