Research map / Detection and segmentation / Open-set and few-shot detection
Gaze Target Detection by Merging Human Attention and Activity Cues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a gaze target detector that fuses a saliency branch with a body-part & object interaction branch, both guided by a soft gaze attention map built from face and head keypoints, reaching SOTA on GazeFollow and VideoAttentionTarget using only 2D input.
Problem
Gaze target detection in intricate image backgrounds, where existing saliency and scene-geometry methods fail to exploit the connection between human attention and activity (body-part & object interaction) cues.
Approach
Combines visual saliency with body-part & object interaction attention, both gated by a soft gaze attention map computed from the face image and five head keypoints (nose, eyes, ears), so gaze target prediction uses activity cues without 3D reconstruction or depth.
Key results
- On GazeFollow, the full method reaches 0.061 min distance and 0.118 avg distance, a 20.8% and 13.9% relative improvement over the Baseline (0.077/0.137).
- On VideoAttentionTarget, the variant without interaction attention (Ours*) reaches 0.106 distance (20.9% lower than Baseline) and 0.910 AP (7.3% higher).
- Compared with Bao et al. 2022 (which uses 3D reconstruction), the method improves average L2 distance by 3.3% using only 2D input; inference is under 75ms per image on a single GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add head keypoints to face-based soft gaze attention (ablation) | Baseline gaze-target detector · GazeFollow test | Min. Dist.: 0.077 -> 0.068 | Helped |
| Add head keypoints to face-based soft gaze attention (ablation) | Baseline gaze-target detector · GazeFollow test | Avg. Dist.: 0.137 -> 0.126 | Helped |
| Add the interaction branch guided by body-part & object interaction attention (ablation) | Ours gaze-target detector · GazeFollow test | Min. Dist.: 0.068 -> 0.061 | Helped |
| Add the interaction branch guided by body-part & object interaction attention (ablation) | Ours gaze-target detector · GazeFollow test | Avg. Dist.: 0.126 -> 0.118 | Helped |
| Use body-part & object interaction attention instead of full-body interaction attention (ablation) | Ours interaction branch · GazeFollow test | Min. Dist.: 0.063 -> 0.061 | Helped |
| Use body-part & object interaction attention instead of full-body interaction attention (ablation) | Ours interaction branch · GazeFollow test | Avg. Dist.: 0.121 -> 0.118 | Helped |
| Add gaze-attention guidance to interaction attention (ablation) | Full-body HOI interaction branch · GazeFollow test | Min. Dist.: 0.066 -> 0.063 | Helped |
| Add gaze-attention guidance to interaction attention (ablation) | Full-body HOI interaction branch · GazeFollow test | Avg. Dist.: 0.124 -> 0.121 | Helped |
| Add gaze- and activity-cue fusion | Baseline gaze-target detector · GazeFollow test | Min. Dist.: 0.077 -> 0.061 | Helped |
| Add gaze- and activity-cue fusion | Baseline gaze-target detector · GazeFollow test | Avg. Dist.: 0.137 -> 0.118 | Helped |
| Add soft gaze attention with head keypoints, without the interaction module (ablation) | Baseline gaze-target detector · VideoAttentionTarget | Dist.: 0.147 -> 0.106 | Helped |
| Add soft gaze attention with head keypoints, without the interaction module (ablation) | Baseline gaze-target detector · VideoAttentionTarget | AP: 0.848 -> 0.910 | Helped |
Surprising findings
- The full model's interaction branch gives no gain on VideoAttentionTarget, which the authors attribute to limited diversity of human activities in that dataset.
Limitations
- Interaction branch offers limited benefit on VideoAttentionTarget due to lack of diverse human activities.
- No activity annotations exist in gaze datasets, so the interaction branch is supervised only through the fused heatmap loss.
- Relies on pre-trained pose estimator and object detector whose errors propagate to the interaction attention.
Experimental setup
Datasets: GazeFollow VideoAttentionTarget
Benchmarks: GazeFollow VideoAttentionTarget
Metrics: Min. L2 distance Avg. L2 distance Out-of-frame AP inference time
Baselines: Baseline (Chong et al. 2020) Chen et al. 2021 Fang et al. 2021 Tu et al. 2022 Bao et al. 2022 Miao et al. 2023
Builds on: Chong et al. 2020 gaze following baseline MobileNet ResNet-50 RTMPose YOLOv3
Closest papers by meaning
- Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.86)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.86)
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.86)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.85)
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.84)
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.