Research map / Architectures and efficiency / Attention and transformers
Sharingan: A Transformer Architecture for Multi-Person Gaze Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sharingan, a transformer that encodes each person as a single location-aware gaze token alongside image tokens and decodes per-person gaze heatmaps with a multiscale Conditional DPT, enabling multi-person gaze following in one forward pass.
Problem
Gaze following (predicting the 2D pixel where a person looks) has been done one person per forward pass with CNNs; prior transformer multi-person methods use fixed learnable queries that require matching, making reliable evaluation and integration hard.
Approach
Each person's head crop is encoded by a gaze backbone into a gaze token (plus a bounding-box embedding) that is concatenated with image tokens in a ViT encoder; a Conditional DPT decoder fuses multi-level image features with each gaze token via element-wise products to produce per-person heatmaps.
Key results
- On GazeFollow, Sharingan reaches AUC 0.944 and Avg. Distance 0.113, beating the multi-person method Jin et al. (0.919 / 0.126) and slightly beating the multimodal single-person method of Gupta et al. (0.114).
- On VideoAttentionTarget, Sharingan obtains Dist. 0.107 and AP 0.891 (PLAH 0.738), state of the art among compared methods.
- On ChildPlay, Sharingan achieves Dist. 0.106 and AP 0.990, improving on Tafasca et al. (0.107 / 0.986).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the location-aware gaze token instead of head-mask embeddings (ablation) | Sharingan · GazeFollow | Avg. D.: 0.117 -> 0.113 | Helped |
| use the location-aware gaze token instead of head-crop tokens (ablation) | Sharingan · GazeFollow | Avg. D.: 0.138 -> 0.113 | Helped |
| use the location-aware gaze token instead of gaze-cone tokens (ablation) | Sharingan · GazeFollow | Avg. D.: 0.133 -> 0.113 | Helped |
| use Conditional DPT rather than direct token-to-heatmap decoding (ablation) | Sharingan · GazeFollow | Avg. D.: 0.302 -> 0.113 | Helped |
| use Conditional DPT rather than token-to-image dot-product decoding (ablation) | Sharingan · GazeFollow | Avg. D.: 0.120 -> 0.113 | Helped |
| upsample the image representation before dot-product decoding (ablation) | Sharingan · GazeFollow | Avg. D.: 0.120 -> 0.116 | Helped |
| increase training people count from N tr_p = 2 to N tr_p = 3 (ablation) | Sharingan · GazeFollow | Avg. D.: 0.113 -> 0.114 | No clear effect |
| increase training people count from N tr_p = 2 to N tr_p = 4 (ablation) | Sharingan · GazeFollow | Avg. D.: 0.113 -> 0.113 | No clear effect |
| remove the angular loss (ablation) | Sharingan · GazeFollow | gaze-following performance | No clear effect |
| regress 2D points instead of predicting gaze heatmaps (ablation) | Sharingan · GazeFollow | Avg. D.: 0.113 -> 0.106 | Helped |
| regress 2D points instead of predicting gaze heatmaps (ablation) | Sharingan · GazeFollow | Min. D.: 0.057 -> 0.066 | Hurt |
Surprising findings
- Adding the angular gaze loss does not change final performance, per the ablations.
- A 2D point regression head gives better average distance and PLAH than the heatmap head but much worse minimum distance and recall of LAH.
Limitations
- Scene and person tokens share the same transformer weights, making it hard to interpret how information is combined.
- Relies on an off-the-shelf head detector and assumes head crops/boxes are available.
- Video predictions may be unstable since the model is image-based.
Experimental setup
Datasets: GazeFollow VideoAttentionTarget ChildPlay CrowdHuman
Benchmarks: GazeFollow VideoAttentionTarget ChildPlay
Metrics: AUC Avg. Distance Min. Distance AP PLAH RLAH
Baselines: Jin et al. multi-person gaze network Tafasca et al. 2023 Gupta et al. gaze following Chong et al. Tu et al. Tonini et al.
Builds on: DPT DETR Multimodal MAE Stochastic Weight Averaging
Closest papers by meaning
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.91)
- Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.89)
- OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.88)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.88)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.87)
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.