Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Few-shot Personalized Scanpath Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes few-shot personalized scanpath prediction (FS-PSP) and ISP-SENet, which uses a subject-embedding network (SE-Net) trained with classification and triplet losses to produce subject embeddings from 1-10 support scanpaths, conditioning a Gazeformer-ISP-based predictor without test-time fine-tuning.
Problem
Predicting an individual's scanpaths from only a few (at most 10) image-scanpath examples of that subject, where existing personalized scanpath models need extensive per-subject data or fine-tuning.
Approach
Decouple subject-embedding learning from scanpath prediction: a separately trained SE-Net extracts subject embeddings from support scanpaths (averaged like prototypical networks) and conditions a frozen personalized predictor, so unseen subjects need no fine-tuning.
Key results
- ISP-SENet outperforms the second-best method on ScanMatch by 5.9% on OSIE, 7.9% on COCO-FreeView, and 6.0% on COCO-Search18 (10-shot setting per abstract/intro).
- On OSIE with 1-shot, ISP-SENet reaches SM 0.368 versus 0.354 for Gazeformer-ISP-S.
- Adaptation for ISP-SENet takes 3.62 seconds versus 161 and 267 seconds of fine-tuning for ChenLSTM-ISP and Gazeformer-ISP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contrastive loss to classification-loss training (ablation) | SE-Net · OSIE | ScanMatch: 0.361 -> 0.375 | Helped |
| add classification loss to contrastive-loss training (ablation) | SE-Net · OSIE | ScanMatch: 0.372 -> 0.375 | Helped |
| add contrastive loss to classification-loss training (ablation) | SE-Net · COCO-Search18 | ScanMatch: 0.448 -> 0.482 | Helped |
| add classification loss to contrastive-loss training (ablation) | SE-Net · COCO-Search18 | ScanMatch: 0.442 -> 0.482 | Helped |
| add the task encoder (ablation) | ISP-SENet · COCO-Search18 | ScanMatch: 0.459 -> 0.482 | Helped |
| add duration information when updating the scanpath embedding (ablation) | ISP-SENet · COCO-Search18 | ScanMatch: 0.446 -> 0.482 | Helped |
| use ISP-SENet instead of the strongest baseline | FS-PSP · OSIE, 1-shot | ScanMatch: 0.354 -> 0.368 | Helped |
| use ISP-SENet instead of the strongest baseline | FS-PSP · COCO-FreeView, 1-shot | ScanMatch: 0.339 -> 0.369 | Helped |
| use ISP-SENet instead of the strongest baseline | FS-PSP · COCO-Search18, 1-shot | ScanMatch: 0.448 -> 0.475 | Helped |
| use ISP-SENet instead of the strongest baseline | FS-PSP · OSIE, 5-shot | ScanMatch: 0.354 -> 0.376 | Helped |
| use ISP-SENet instead of the strongest baseline | FS-PSP · COCO-FreeView, 5-shot | ScanMatch: 0.338 -> 0.368 | Helped |
| use ISP-SENet instead of the strongest baseline | FS-PSP · COCO-Search18, 5-shot | ScanMatch: 0.449 -> 0.484 | Helped |
Surprising findings
- Fine-tuning only the subject embedding layer (ISP-S variants) performs better than fine-tuning all parameters, which the authors attribute to similarity between seen and unseen subjects.
- Unseen-subject embeddings from SE-Net nearly match the fully trained upper bound on OSIE and COCO-Search18 without training on those subjects.
Limitations
- SE-Net may overemphasize unique subject traits while overlooking shared patterns useful for scanpath prediction.
- Distinguishing unseen from seen subjects is hard given the small subject pool, potentially biasing toward known subjects.
- Performance on COCO-FreeView with unseen subjects trails the fully-trained seen-subject model.
Experimental setup
Datasets: OSIE COCO-FreeView COCO-Search18
Benchmarks: OSIE COCO-FreeView COCO-Search18
Metrics: ScanMatch MultiMatch String-Edit Distance scanpath accuracy (R@1)
Baselines: ChenLSTM-ISP Gazeformer-ISP ChenLSTM-ISP-S Gazeformer-ISP-S
Builds on: Gazeformer-ISP prototypical networks HAT ISP
Closest papers by meaning
- Beyond Average: Individualized Visual Scanpath Prediction (CVPR 2024 · similarity 0.88)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.84)
- SpFormer: Spatio-Temporal Modeling for Scanpaths with Transformer (AAAI 2024 · similarity 0.83)
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.83)
- DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images (NeurIPS 2025 · similarity 0.83)
- DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.