Research map / Detection and segmentation / Open-set and few-shot detection
Toward Semantic Gaze Target Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends gaze following to semantic gaze target detection by jointly predicting gaze heatmap and open-vocabulary CLIP-aligned class label, with pseudo-labeled GazeFollow and a new GazeHOI benchmark.
Problem
Gaze following methods only predict where a person looks (pixel location), not what the gaze target is semantically; existing semantic alternatives rely on post-hoc detectors or HOI datasets with limited vocabularies.
Approach
A promptable, late-fusion transformer decoder that encodes the scene once and decodes per-person gaze tokens into a heatmap and a gaze-label embedding aligned to frozen CLIP text embeddings via a contrastive loss over unique batch labels, trained on pseudo-labeled GazeFollow.
Key results
- Sets new state-of-the-art localization on GazeFollow with Avg Dist 0.108 and Min Dist 0.051, beating prior best Tafasca et al. at 0.113/0.057.
- Recognition on GazeFollow reaches Acc@1 0.447 and Acc@3 0.642, above frozen CLIP-crop baseline (0.239) and close to heatmap-weighted baseline (0.466), with 116M vs 200M parameters.
- On GazeHOI, Acc@1 0.583 vs 0.646 for the fine-tuned heatmap-weighted baseline; zero-shot baselines are much lower than the proposed model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Tafasca et al. [49] with the proposed model | Gaze target localization on GazeFollow · GazeFollow test | Avg. Dist.: 0.113 -> 0.108 | Helped |
| use the proposed semantic model instead of heatmap-weighted CLIP baseline | Gaze target recognition · GazeFollow test | Acc@1: 0.466 -> 0.447 | Hurt |
| use the proposed semantic model instead of heatmap-weighted CLIP baseline | Gaze target recognition · GazeHOI test | Acc@1: 0.646 -> 0.583 | Hurt |
| increase batch size from 72 to 216 (ablation) | Ours on GazeFollow · GazeFollow | Acc@1: 0.405 -> 0.450 | Helped |
| increase batch size from 216 to 300 (ablation) | Ours on GazeFollow · GazeFollow | Acc@1: 0.450 -> 0.447 | No clear effect |
| remove heatmap localization loss and train with label loss only (ablation) | Ours on GazeFollow · GazeFollow | Acc@1: 0.451 -> 0.417 | Hurt |
| add label loss to heatmap loss (ablation) | Ours on GazeFollow · GazeFollow | Avg. D.: 0.107 -> 0.108 | No clear effect |
| add angular loss to heatmap and label losses (ablation) | Ours on GazeFollow · GazeFollow | Acc@1: 0.451 -> 0.447 | No clear effect |
| increase the number of people during training from 1 to 3 or 5 (ablation) | Ours on GazeFollow · GazeFollow | Avg. D.: 0.108 -> 0.112; 0.108 -> 0.113 | Hurt |
| use bounding-box coordinates only instead of both box coordinates and head crop (ablation) | Gaze encoder on GazeFollow · GazeFollow | Avg. D.: 0.107 -> 0.132 | Hurt |
| use head crop only instead of both box coordinates and head crop (ablation) | Gaze encoder on GazeFollow · GazeFollow | Avg. D.: 0.107 -> 0.133 | Hurt |
| replace MultiMAE image pretraining with CLIP vision pretraining (ablation) | Image encoder on GazeFollow · GazeFollow | Avg. Dist.: 0.108 -> 0.148 | Hurt |
Surprising findings
- Random projections as text embeddings perform on par with CLIP text embeddings on GazeFollow, showing the text encoder choice has little effect in-domain.
- Using CLIP vision weights for both localization and recognition degrades localization substantially (Avg Dist 0.148 vs 0.108).
Limitations
- Predicts a single gaze label embedding, so it cannot handle multi-label gaze targets well.
- GazeHOI recognition is weaker than fine-tuned baselines, attributed to small dataset scale and label mismatch between pseudo-labels and GazeHOI vocabulary.
- Pseudo-labels from open-world segmentation are noisy and the test set was annotated by a single person; no error bars reported.
Experimental setup
Datasets: GazeFollow V-COCO HICO-DET HCVRD SWiG-HOI HOI-A Gaze360
Benchmarks: GazeFollow GazeHOI
Metrics: Avg Dist Min Dist Acc@1 Acc@3 MultiAcc@1 GazeAcc
Baselines: Chong et al. [10] Fang et al. [16] Jin et al. [27] Tonini et al. [50] Tafasca et al. [49] Baseline (Mask/Blur/Crop/heatmap weight) CLIP-based
Builds on: promptable segmentation (SAM) CLIP MultiMAE DARK coordinate decoding
Closest papers by meaning
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.91)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.89)
- OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.89)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.89)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.87)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.