AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Look Hear: Gaze Prediction for Speech-directed Human Attention

Sounak Mondal, Seoyoung Ahn, Zhibo Yang, Niranjan Balasubramanian, Dimitris Samaras, Gregory Zelinsky, Minh Hoai

ECCV 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Human perception modeling · 8 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
pre-train ART on RefCOCO grounding objectives (with both grounding losses) (ablation)ART · RefCOCO-Gaze testSequence Score (SS): 0.304 -> 0.359Improved
add both L_bbox and L_target grounding losses in pre-training and training (ablation)ART · RefCOCO-Gaze testSequence Score (SS): 0.321 -> 0.359Improved
add grounding losses without pre-training (training from scratch) (ablation)ART · RefCOCO-Gaze testSequence Score (SS): 0.309 -> 0.304Hurt
include only one grounding loss (L_target only) with pre-training (ablation)ART · RefCOCO-Gaze testSequence Score (SS): 0.309 -> 0.292Hurt
include only L_bbox with pre-training (ablation)ART · RefCOCO-Gaze testSequence Score (SS): 0.309 -> 0.321No clear effect
use ART (joint grounding, autoregressive pack decoder) instead of Gazeformer-ref baselinescanpath prediction · RefCOCO-Gaze testSequence Score (SS): 0.269 -> 0.359Improved
give Gazeformer only the estimated target category instead of the expression prefixGazeformer · RefCOCO-Gaze testSequence Score (SS): 0.269 -> 0.269No clear effect
apply ART to categorical search on COCO-Search18 instead of the Gazeformer baselineART · COCO-Search18 (target-present)Sequence Score (SS): 0.475 -> 0.454Mixed
train ART without any pre-training or grounding losses still beats Gazeformer baselines (ablation)ART · RefCOCO-Gaze testSequence Score (SS): 0.269 -> 0.309Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy