Research map / Vision-language and multimodal / Human and embodied modeling
Beyond Average: Individualized Visual Scanpath Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes individualized scanpath prediction (ISP), adding an observer encoder, observer-centric feature integration, and adaptive fixation prioritization to general scanpath models (ChenLSTM, Gazeformer) to predict how specific observers move their gaze.
Problem
Existing visual scanpath models predict observer-agnostic gaze sequences and ignore individual differences in attention across observers, including differences linked to demographics and ASD.
Approach
Encode each observer's identity as a learned embedding and inject it into visual-feature fusion and output prioritization so a general encoder-decoder scanpath model becomes observer-specific, rather than fine-tuning per observer.
Key results
- ChenLSTM-ISP raises R@1 on OSIE-ASD from 2.4% (ChenLSTM) to 4.8%, and MRR on OSIE from 0.222 to 0.291.
- Gazeformer-ISP reaches SM=0.406 on OSIE-ASD versus 0.388 for Gazeformer, and ChenLSTM-ISP reaches SM=0.480 on COCO-Search18 versus 0.454 for ChenLSTM.
- ISP models classify ASD vs control from learned observer features with 82.1% accuracy (ChenLSTM-ISP) and 71.8% (Gazeformer-ISP) under leave-one-out cross-validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply individualized scanpath prediction (ISP) instead of fine-tuning | ChenLSTM · OSIE-ASD | SM: 0.394 -> 0.401 | Helped |
| apply individualized scanpath prediction (ISP) instead of fine-tuning | ChenLSTM · COCO-Search18 | SM: 0.454 -> 0.480 | Helped |
| apply individualized scanpath prediction (ISP) instead of fine-tuning | ChenLSTM · AiR-D | SM: 0.341 -> 0.371 | Helped |
| add observer encoder (ablation) | ChenLSTM on OSIE-ASD · OSIE-ASD | SM: 0.341 -> 0.377 | Helped |
| add observer-centric feature integration (ablation) | ChenLSTM with observer encoder on OSIE-ASD · OSIE-ASD | MRR: 0.110 -> 0.122 | Helped |
| add adaptive fixation prioritization (ablation) | ChenLSTM with observer encoder on OSIE-ASD · OSIE-ASD | MRR: 0.110 -> 0.112 | No clear effect |
| add observer-centric feature integration (ablation) | Gazeformer with observer encoder on OSIE-ASD · OSIE-ASD | MRR: 0.122 -> 0.134 | Helped |
| add adaptive fixation prioritization (ablation) | Gazeformer with observer encoder on OSIE-ASD · OSIE-ASD | MRR: 0.122 -> 0.120 | No clear effect |
| add both observer-centric feature integration and adaptive fixation prioritization (ablation) | ChenLSTM with observer encoder on OSIE-ASD · OSIE-ASD | MRR: 0.110 -> 0.147 | Helped |
| add both observer-centric feature integration and adaptive fixation prioritization (ablation) | Gazeformer with observer encoder on OSIE-ASD · OSIE-ASD | MRR: 0.122 -> 0.141 | Helped |
Surprising findings
- Fine-tuning general models on individual observer data yields only marginal gains and sometimes lower-than-random R@1 for general models, showing observer-agnostic models cannot distinguish observers.
Limitations
- Gains on datasets with less distinct inter-observer differences (COCO-Search18, AiR-D) are small, and fine-tuning helps little there.
- Observer pools are small (e.g., 39 observers in OSIE-ASD, 10 in COCO-Search18), limiting generalization claims.
- ASD classification accuracy is evaluated on a small cohort and is not validated clinically.
Experimental setup
Datasets: OSIE OSIE-ASD COCO-Search18 AiR-D
Benchmarks: OSIE OSIE-ASD COCO-Search18 AiR-D
Metrics: ScanMatch MultiMatch String-Edit Distance Recall at K Mean Reciprocal Rank CC AUC NSS sAUC KLD
Baselines: SaltiNet PathGAN IOR-ROI ChenLSTM Gazeformer ChenLSTM-FT Gazeformer-FT
Builds on: ChenLSTM Gazeformer ScanMatch MultiMatch
Closest papers by meaning
- Few-shot Personalized Scanpath Prediction (CVPR 2025 · similarity 0.88)
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.87)
- Attention IoU: Examining Biases in CelebA using Attention Maps (CVPR 2025 · similarity 0.83)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.83)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.83)
- DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.