Research map / Large language models / Language, safety and interpretability
Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention
Sounak Mondal, Dimitris Samaras, Gregory Zelinsky, Minh Hoai Nguyen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ThinkAloud auxiliary task (ablation) | Gazette · RefCOCO-Gaze object referral (Task A) | CIDEr: 0.872 -> 0.974 | Improved |
| add ThinkAloud auxiliary task (ablation) | Gazette · RefCOCO-Gaze object referral (Task A) | BLEU-4: 0.070 -> 0.098 | Improved |
| add ThinkAloud auxiliary task (ablation) | Gazette · AiR-D VQA (Task B) | CIDEr: 0.263 -> 0.367 (+39.54%) | Improved |
| add ThinkAloud auxiliary task (ablation) | Gazette · AiR-D VQA (Task B) | BLEU-4: 0.095 -> 0.159 (+67.37%) | Improved |
| add ThinkAloud auxiliary task (ablation) | Gazette · RefCOCO-Gaze object referral (Task A) | GPT-4 judge Expression Overlap: 5.019 -> 5.980 | Improved |
| add ThinkAloud auxiliary task (ablation) | Gazette · AiR-D VQA (Task B) | GPT-4 judge Answer Equivalence: 1.642 -> 2.135 | Improved |
| add ThinkAloud auxiliary task (ablation) | Gazette · COCO-Search18 target-present search (Task C) | accuracy: 0.742 -> 0.773 | Improved |
| add ThinkAloud auxiliary task (ablation) | Gazette · COCO-Search18 target-absent search (Task D) | accuracy: 0.445 -> 0.443 | No clear effect |
| add LLaVA-last style last-fixation baseline instead of full scanpath | LLaVA-1.5-7B · RefCOCO-Gaze object referral (Task A) | CIDEr: 0.974 -> 0.113 | Hurt |
| add attention allocation explanation (GenAttAlloc) alone vs target localization and fixation count (full set) (ablation) | Gazette · RefCOCO-Gaze object referral (Task A) | CIDEr: 0.838 -> 0.974 | Improved |
| add LocTarget and CountFix to GenAttAlloc-only transcript (ablation) | Gazette · AiR-D VQA (Task B) | CIDEr: 0.433 -> 0.367 | Hurt |
| add LocTarget and CountFix to GenAttAlloc-only transcript (ablation) | Gazette · COCO-Search18 target-present search (Task C) | accuracy: 0.686 -> 0.773 | Improved |
Closest papers by meaning
- OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.89)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.88)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.88)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.87)
- CoFFT: Chain of Foresight-Focus Thought for Visual Language Models (NeurIPS 2025 · similarity 0.87)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
