Research map / Image generation / Fast diffusion sampling
DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffEye, a DDPM/DDIM-based diffusion model trained on raw MIT1003 eye-tracking trajectories conditioned on DINOv2+FeatUp patch features via a Corresponding Positional Embedding, generating trajectories convertible to scanpaths and saliency maps.
Problem
Scanpath and saliency prediction models usually train on discretized scanpaths, discard raw trajectories, and predict a single deterministic output, failing to capture inter-subject variability in free viewing of natural images.
Approach
A diffusion model operating directly on continuous raw eye-movement trajectories, conditioned on high-resolution patch-level image features through cross-attention at every U-Net block, with a Corresponding Positional Embedding that shares a 2D sinusoidal grid between gaze coordinates and image patches.
Key results
- On MIT1003, DiffEye achieves mean Levenshtein 13.009 vs 14.415 for DeepGaze III (seen) and mean TDE 88.661 vs 92.100 for Chen et al.
- On unseen OSIE, DiffEye gets best mean Levenshtein 14.771 and DTW 1.552 (x10^3), outperforming HAT and IOR-ROI on mean metrics.
- Saliency maps derived from DiffEye trajectories underperform dedicated saliency models (e.g., NSS 1.991 vs 3.611 for SUM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · MIT1003 | Levenshtein Distance (mean): 13.574 -> 13.009 | Helped |
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · OSIE | Levenshtein Distance (mean): 14.836 -> 14.771 | Helped |
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · MIT1003 | Discrete Fréchet Distance (mean): 3.553 -> 3.529 | Helped |
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · OSIE | Discrete Fréchet Distance (mean): 3.206 -> 3.068 | Helped |
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · MIT1003 | Dynamic Time Warping (mean): 1.834 -> 1.573 | Helped |
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · OSIE | Dynamic Time Warping (mean): 1.699 -> 1.552 | Helped |
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · MIT1003 | Time Delay Embedding (mean): 92.100 -> 88.661 | Helped |
| Train DiffEye on raw eye movement trajectories and generate scanpaths | DiffEye · OSIE | Time Delay Embedding (mean): 78.286 -> 81.925 | Hurt |
| Add FeatUp high-resolution patch features (ablation) | DiffEye · Scanpath Generation | Levenshtein Distance (mean): 0.133 -> 0.130 | Helped |
| Add CPE (ablation) | DiffEye · Scanpath Generation | Levenshtein Distance (mean): 0.141 -> 0.130 | Helped |
| Add cross-attention at U-Net blocks (ablation) | DiffEye · Scanpath Generation | Levenshtein Distance (mean): 0.143 -> 0.130 | Helped |
| Use patch-level features instead of a global token (ablation) | DiffEye · Scanpath Generation | Levenshtein Distance (mean): 0.153 -> 0.130 | Helped |
Surprising findings
- Saliency maps derived from generated trajectories are weaker than dedicated saliency models on most metrics, with KL divergence of 1.991 versus 0.552 for DeepGaze IIE.
Limitations
- Trained only on the small MIT1003 dataset (8,934 trajectories), limiting scaling.
- Generates only fixed-length outputs at 240 Hz; cannot produce variable-length sequences.
- Only free-viewing task is covered because only FV raw-trajectory data is available.
Experimental setup
Datasets: MIT1003 OSIE
Benchmarks: MIT1003 OSIE
Metrics: Levenshtein distance discrete Fréchet distance dynamic time warping time delay embedding AUC-Judd AUC-Borji NSS SIM CC KL
Baselines: IOR-ROI DeepGaze III Chen et al. GazeFormer HAT SUM TranSalNet DeepGaze IIE
Builds on: DDPM DDIM classifier-free guidance DINOv2 FeatUp
Closest papers by meaning
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.93)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.87)
- GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction (NeurIPS 2026 · similarity 0.87)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.87)
- EyeGraph: Modularity-aware Spatio Temporal Graph Clustering for Continuous Event-based Eye Tracking (NeurIPS 2024 · similarity 0.86)
- CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.