Research map / Video / Video segmentation and tracking
MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTGS, a temporal multi-person transformer that jointly predicts gaze targets and social gaze labels (LAH, LAEO, SA) for all people in a video, trained on a unified dataset VSGaze built from four video gaze datasets.
Problem
Gaze following and social gaze prediction are usually handled separately, by single-person static models or by ad-hoc post-processing, and video/temporal multi-person joint modeling has been underexplored due to limited unified annotations.
Approach
Person-specific tokens interact with image tokens through people-scene and spatio-temporal social interaction blocks, with a person-conditioned DPT heatmap decoder and pairwise decoders for LAH, LAEO and SA, all trained jointly on a unified VSGaze dataset with newly derived social gaze annotations.
Key results
- On VSGaze, the model (Ours) reaches distance 0.111 and AP_SA 0.577, versus 0.121 distance and 0.335 AP_SA for the best re-trained baseline (Gupta*).
- On UCO-LAEO after fine-tuning, the model reaches AP_LAEO 0.974, far above prior LAEO-specific methods (best 0.867).
- On GazeFollow, the static model achieves the best average and minimum distances (0.118 and 0.062) among compared methods, and on VAT it reaches distance 0.110 (best among multi-person methods).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add social gaze losses (ablation) | MTGS · VSGaze | AP_SA: 0.515 -> 0.577 | Helped |
| add social gaze losses (ablation) | MTGS · VAT | AP_SA: 0.443 -> 0.521 | Helped |
| add gaze following losses (ablation) | MTGS · VSGaze | AP_SA: 0.515 -> 0.577 | Helped |
| use a temporal window of 5 frames instead of 1 (ablation) | MTGS · VSGaze | AP_SA: 0.555 -> 0.577 | Helped |
| use a temporal window of 9 instead of 5 frames (ablation) | MTGS · VSGaze | AP_SA: 0.577 -> 0.563 | Hurt |
| add Person-to-Scene encoder (ablation) | MTGS · VSGaze | Dist.: 0.115 -> 0.111 | Helped |
| add Person-to-Scene encoder (ablation) | MTGS · VSGaze | AP_SA: 0.538 -> 0.577 | Helped |
| add Scene-to-Person encoder (ablation) | MTGS · VSGaze | Dist.: 0.107 -> 0.111 | Hurt |
| add Scene-to-Person encoder (ablation) | MTGS · VSGaze | AP_SA: 0.545 -> 0.577 | Helped |
| add Spatio-Temporal Social Interaction component (ablation) | MTGS · VSGaze | AP_SA: 0.547 -> 0.577 | Helped |
| use modified DPT instead of the simpler heatmap decoder (ablation) | MTGS · VSGaze | AP_SA: 0.528 -> 0.577 | Helped |
| add predicted speaking status (ablation) | MTGS fine-tuned on ChildPlay · ChildPlay | F1_LAEO: 0.431 -> 0.450 | Helped |
Surprising findings
- Temporal baseline Chong_T often performs worse than its static counterpart Chong_S, with lower distance and LAEO scores.
- Post-processing gaze following outputs for LAH and LAEO gives slightly better results than the dedicated social decoders, while the SA decoder is clearly better than post-processing.
Limitations
- Temporal modeling gains are modest and not captured well by standard metrics since gaze shifts are a small fraction of instances.
- Joint training on VSGaze underperforms dataset-specific fine-tuning because of differing dataset priors.
- Speaking-status detector transfers poorly to ChildPlay, limiting the auxiliary-information gains.
Experimental setup
Datasets: GazeFollow VideoAttentionTarget ChildPlay VideoCoAtt UCO-LAEO Gaze360
Benchmarks: GazeFollow VideoAttentionTarget ChildPlay VideoCoAtt UCO-LAEO VSGaze
Metrics: AUC Avg distance Min distance AP IO F1 LAH F1 LAEO AP SA
Baselines: Chong (static) Chong (temporal) Gupta et al. gaze following Jin et al. Tafasca et al. Fang et al. Jiminez et al.
Builds on: DPT MultiMAE ViT DETR
Closest papers by meaning
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.91)
- Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.91)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.90)
- OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.90)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.87)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.