Research map / Architectures and efficiency / Attention and transformers
Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HAT, a transformer model with a simplified foveated working memory that predicts dense fixation heatmaps and termination for both visual search and free-viewing scanpaths.
Problem
Scanpath prediction models have usually targeted either top-down (visual search) or bottom-up (free-viewing) attention and rely on coarse fixation grids, losing precision.
Approach
A transformer that keeps a dynamic working memory of peripheral tokens (low-res P1) and foveal tokens (high-res P4 at previous fixations), with task-specific queries, and predicts dense per-pixel fixation heatmaps plus termination probabilities without discretizing fixations.
Key results
- On COCO-Search18 target-present, HAT reaches cNSS 5.086 and SemSS 0.543 versus 2.606 cNSS for Chen et al.
- On COCO-Search18 target-absent, HAT achieves SS 0.402, surpassing human consistency (0.381), and cNSS 3.103 vs 1.600 for the prior best.
- On COCO-FreeView, HAT reaches cNSS 3.382 vs 1.655 for the second best (Chen et al.), roughly 104% improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HAT instead of Chen et al. | Scanpath prediction · COCO-Search18 target-present test | cNSS: 2.606 -> 5.086 | Helped |
| use HAT instead of Chen et al. | Scanpath prediction · COCO-Search18 target-absent test | cNSS: 1.600 -> 3.103 | Helped |
| use HAT instead of Chen et al. | Scanpath prediction · COCO-FreeView test | cNSS: 1.655 -> 3.382 | Helped |
| add peripheral tokens (ablation) | HAT · COCO-Search18 target-absent test | cNSS: 3.003 -> 3.103 | Helped |
| add foveal tokens (ablation) | HAT · COCO-Search18 target-absent test | cNSS: 2.380 -> 3.103 | Helped |
| reduce output resolution to 20×32 (ablation) | HAT · COCO-Search18 target-absent test | cNSS: 3.103 -> 2.760 | Hurt |
Surprising findings
- HAT's sequence score surpasses human consistency in target-absent search for the first time.
- Removing peripheral tokens causes only a minor performance drop, while removing foveal tokens causes a large drop.
Limitations
- Fixations are rounded to the nearest P4 position for foveal tokens, a simplification of the retina.
- Scanpaths are generated greedily by selecting the most probable fixation, producing one scanpath per image.
- Target-absent and free-viewing predictions remain harder than target-present, with large gaps to human consistency in some metrics.
Experimental setup
Datasets: COCO-Search18 COCO-FreeView MIT1003 OSIE
Benchmarks: COCO-Search18 target-present COCO-Search18 target-absent COCO-FreeView
Metrics: SS SemSS cIG cNSS cAUC
Baselines: Human consistency Detector Fixation heuristic IVSN PathGAN IRL Chen et al. DeepGaze III FFMs Gazeformer
Builds on: Transformer Gazeformer FFMs Chen et al. scanpath RL model MSDeformAttn FPN
Closest papers by meaning
- Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.87)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.87)
- Beyond Average: Individualized Visual Scanpath Prediction (CVPR 2024 · similarity 0.87)
- Learning from Observer Gaze: Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition (CVPR 2024 · similarity 0.87)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.86)
- SpFormer: Spatio-Temporal Modeling for Scanpaths with Transformer (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.