Research map / Vision-language and multimodal / Multimodal LLMs
The Language of Visual Attention: Modeling Scanpaths via Autoregressive Token Prediction
Susmit Agrawal, Matthias Bethge, Matthias Kuemmerer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune a VLM for autoregressive coordinate-token scanpath prediction instead of DeepGaze III | DeepGaze3.5-VL · MIT1003 | Information Gain: 1.49 -> 2.18 | Improved |
| give DeepGaze III the larger InternViT-6B encoder (ablation) | DeepGaze III · MIT1003 | Information Gain: 1.45 -> 1.49 | Improved |
| add duration conditioning to the tokens (ablation) | DeepGaze3.5-VL · MIT1003 | Information Gain: 2.03 -> 2.12 | Improved |
| add duration conditioning to the tokens (ablation) | DeepGaze3.5-VL · COCO-FreeView | Information Gain: 2.15 -> 2.30 | Improved |
| add duration conditioning to the tokens (ablation) | DeepGaze3.5-VL · DAEMONS | Information Gain: 2.67 -> 2.84 | Improved |
| add subject identity token to the prompt (ablation) | DeepGaze3.5-VL · MIT1003 (leave-subject-out) | Information Gain: +0.09 bits/fix | Improved |
| put the search target in the prompt | DeepGaze3.5-VL · COCO-Search18 | Information Gain: 2.15 -> 3.27 | Improved |
| search conditioning when target is absent (ablation) | DeepGaze3.5-VL · COCO-Search18 (target absent) | Information Gain: 2.29 vs 2.15 (free viewing) | No clear effect |
| train on all five datasets combined instead of in-distribution only (ablation) | DeepGaze3.5-VL · MIT1003 | Information Gain: 2.03 -> 2.18 | Improved |
| train on all datasets including the target vs leave-one-dataset-out (ablation) | DeepGaze3.5-VL · MIT1003 | Information Gain: 1.94 -> 2.18 | Improved |
| stronger multimodal backbone (higher MMMU) (ablation) | LoRA-tuned VLMs · MIT1003 | Information Gain: Spearman rho = 0.93 | Improved |
| scale SmolVLM2 backbone from 256M to 2.2B (ablation) | SmolVLM2 · MIT1003 | Information Gain: 0.85 bits across scales | No clear effect |
Closest papers by meaning
- Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding (CVPR 2026 · similarity 0.89)
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.89)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.89)
- Rethinking Causal Mask Attention for Vision-Language Inference (ICLR 2026 · similarity 0.89)
- Each Complexity Deserves a Pruning Policy (NeurIPS 2025 · similarity 0.89)
- Pathways of Visual Information Flow in Vision-Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
