Research map / Video / Video segmentation and tracking
SpFormer: Spatio-Temporal Modeling for Scanpaths with Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpFormer, a Transformer for saccadic scanpaths that uses fixation-centric patch tokenization, local meta attention with a consistency loss, and progressive duration aggregation, evaluated on ASD recognition, toddler age prediction, and visual perceptual task prediction.
Problem
Scanpath models suffer from spatial misalignment of fixation features and lose temporal correlation and fixation duration information, so they fail to give effective spatio-temporal representations of eye-tracking scanpaths.
Approach
Crop image patches centered on each fixation to get spatially aligned fixation tokens, apply a causal global temporal mask plus a local meta attention inspired by visual working memory limits (with a consistency loss), and progressively inject fixation duration as decaying weights across Transformer blocks.
Key results
- On Saliency4ASD subject-wise, SpFormer reaches 100% AUC, sensitivity, specificity, BA and 0.5-threshold accuracy.
- On the in-house ASD dataset, subject-wise accuracy at 0.5 threshold rises from 86.90% to 93.14% over the prior best.
- On the VPT free-viewing vs. cued object search task, SpFormer reaches 0.9974 AUC and 97.5% accuracy at 0.5 threshold.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SpFormer instead of the strongest reported baseline | ASD recognition · Saliency4ASD, subject-wise | AUC: 0.9286 -> 1.0000 | Helped |
| use SpFormer instead of the strongest reported baseline | ASD recognition · Collected ASD dataset, subject-wise | accuracy at 0.5 threshold: 0.8690 -> 0.9314 | Helped |
| use SpFormer instead of the strongest reported baseline | toddler age prediction · TAP | accuracy at 0.5 threshold: 0.8300 -> 0.8750 | Helped |
| use SpFormer instead of the strongest reported baseline | visual perceptual task prediction · VPT, free-viewing vs cued object search | accuracy at 0.5 threshold: 0.8635 -> 0.9750 | Helped |
| add aligned spatial fixation features instead of misaligned fixation features (ablation) | SpFormer · Saliency4ASD, subject-wise | accuracy at 0.5 threshold: 0.7433 -> 0.9000 | Helped |
| add fixation temporal correlation modeling (ablation) | SpFormer with aligned spatial fixation features · Saliency4ASD, subject-wise | AUC: 0.8901 -> 0.9835 | Helped |
| add progressive duration aggregation (ablation) | SpFormer with aligned spatial fixation features and temporal correlation · Saliency4ASD, subject-wise | accuracy at 0.5 threshold: 0.9333 -> 0.9667 | Helped |
| add consistency loss Lmeta (ablation) | SpFormer with aligned spatial fixation features, temporal correlation, and duration aggregation · Saliency4ASD, subject-wise | AUC: 0.9835 -> 1.0000 | Helped |
| vary progressive duration decay coefficient σ across 0.1, 0.3, 0.5, 0.7, and 0.9 (ablation) | SpFormer · Saliency4ASD, subject-wise | AUC: 0.9615, 0.9890, 1.0000, 0.9945, 0.9835 | Mixed |
| vary consistency-loss coefficient λ across 0.01, 0.05, 0.1, 0.2, and 0.5 (ablation) | SpFormer · Saliency4ASD, subject-wise | AUC: 0.9945, 0.9890, 1.0000, 0.9560, 0.9505 | Mixed |
Limitations
- Ablations and hyperparameter study are run only on Saliency4ASD with small sample sizes.
- Several baselines are reported from prior papers rather than re-run under the same protocol.
Experimental setup
Datasets: Saliency4ASD in-house ASD eye-tracking dataset TAP dataset (Dalrymple et al. 2019) VPT dataset (Koehler et al. 2014)
Benchmarks: Saliency4ASD in-house ASD dataset TAP VPT
Metrics: AUC sensitivity specificity balanced accuracy accuracy
Baselines: DoF APM RM3ASD SSM IBM SySM SP-ASDNET CETS HoG Gist VGG16 CAET
Builds on: Vision Transformer ViT class token training
Closest papers by meaning
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.86)
- DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images (NeurIPS 2025 · similarity 0.84)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.84)
- Few-shot Personalized Scanpath Prediction (CVPR 2025 · similarity 0.83)
- SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers (CVPR 2025 · similarity 0.83)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.