Research map / Video / Video generation
ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST-DiffEye, a diffusion transformer that jointly generates continuous eye trajectories and scanpaths by appending a fixation-index channel, and a CRPS-based distributional evaluation protocol for gaze generation.
Problem
Generating human gaze (scanpaths and continuous trajectories) under free-viewing and visual search, where gaze is highly variable across viewers, and existing evaluations (mean, best, KLD) fail to measure distributional accuracy and diversity.
Approach
Augment each trajectory token with a continuous fixation-index channel derived by aligning fixations to trajectory points, so one diffusion model jointly denoises both modalities; and wrap any sequence similarity metric in a CRPS proper scoring rule for evaluation.
Key results
- On COCO-Search18 target-present, ST-DiffEye reaches DFD 110.038 and DTW 196.631 versus ScanDiff's 124.455 and 219.855 under the CRPS protocol.
- On MIT1003 scanpath CRPS, ST-DiffEye ranks first on 5 of 9 metrics, including SS-dagger 0.146 vs ScanDiff 0.128.
- Ablation on COCO-Search18 TP: raw-channel fixation-index concatenation gives the best results, while embedding-space conditioning substantially degrades performance (e.g., DFD 323.933).
Surprising findings
- Under the mean and best-of-N protocols, ST-DiffEye is not best on most metrics, e.g., ScanDiff or GazeFormer win several mean-protocol columns.
- Latent-space fixation-index injection works better than raw-channel injection on MIT1003, attributed to noisy trajectory annotations.
Limitations
- Trajectory annotations in MIT1003 are noisy, with trajectories moving out of frame, reducing trajectory-scanpath alignment.
- Separate models are trained per dataset rather than a unified model.
Experimental setup
Datasets: COCO-Search18 COCO-FreeView MIT1003
Benchmarks: COCO-Search18 COCO-FreeView MIT1003
Metrics: LD DFD DTW TDE MultiMatch ScanMatch Sequence Score Semantic Sequence Score CRPS KLD
Baselines: GazeFormer GazeXplain HAT TPP-Gaze DeepGaze III DiffEye ScanDiff
Builds on: ScanDiff DiffEye DDPM CRPS
Closest papers by meaning
- DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images (NeurIPS 2025 · similarity 0.93)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.90)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.90)
- Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.87)
- OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.87)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.