Research map / Vision-language and multimodal / Human and embodied modeling
Look Hear: Gaze Prediction for Speech-directed Human Attention
Sounak Mondal, Seoyoung Ahn, Zhibo Yang, Niranjan Balasubramanian, Dimitris Samaras, Gregory Zelinsky, Minh Hoai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train ART on RefCOCO grounding objectives (with both grounding losses) (ablation) | ART · RefCOCO-Gaze test | Sequence Score (SS): 0.304 -> 0.359 | Improved |
| add both L_bbox and L_target grounding losses in pre-training and training (ablation) | ART · RefCOCO-Gaze test | Sequence Score (SS): 0.321 -> 0.359 | Improved |
| add grounding losses without pre-training (training from scratch) (ablation) | ART · RefCOCO-Gaze test | Sequence Score (SS): 0.309 -> 0.304 | Hurt |
| include only one grounding loss (L_target only) with pre-training (ablation) | ART · RefCOCO-Gaze test | Sequence Score (SS): 0.309 -> 0.292 | Hurt |
| include only L_bbox with pre-training (ablation) | ART · RefCOCO-Gaze test | Sequence Score (SS): 0.309 -> 0.321 | No clear effect |
| use ART (joint grounding, autoregressive pack decoder) instead of Gazeformer-ref baseline | scanpath prediction · RefCOCO-Gaze test | Sequence Score (SS): 0.269 -> 0.359 | Improved |
| give Gazeformer only the estimated target category instead of the expression prefix | Gazeformer · RefCOCO-Gaze test | Sequence Score (SS): 0.269 -> 0.269 | No clear effect |
| apply ART to categorical search on COCO-Search18 instead of the Gazeformer baseline | ART · COCO-Search18 (target-present) | Sequence Score (SS): 0.475 -> 0.454 | Mixed |
| train ART without any pre-training or grounding losses still beats Gazeformer baselines (ablation) | ART · RefCOCO-Gaze test | Sequence Score (SS): 0.269 -> 0.309 | Improved |
Closest papers by meaning
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.90)
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.88)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.88)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.88)
- Learning from Observer Gaze: Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition (CVPR 2024 · similarity 0.87)
- A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
