Research map / Video / Video segmentation and tracking
FIFA: Fine-grained Inter-frame Attention for Driver's Video Gaze Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FIFA, a fine-grained inter-frame cross-attention module plus a parallel CNN-Transformer dual-stream network for driver video gaze estimation, achieving state-of-the-art angular error on IV and LBW with fewer parameters.
Problem
Video-based 3D gaze estimation for drivers, where pupil displacement across frames must be modeled while head pose from a relatively static background must be preserved, under tight in-vehicle parameter and compute budgets.
Approach
Partition low-level face features into patches, compute inter-frame feature differences as queries and cross-attend to current-frame patches to highlight pupil displacement; the resulting attention map gates Transformer gaze embeddings in a parallel dual-stream design.
Key results
- On IV, FIFA reaches 7.29° mean angular error vs. 7.33° GazePTR, 7.31° GazeTR and 8.05° ResNet18.
- On LBW, FIFA reaches 5.84° error vs. 6.11° STAGE and 6.16° GazePTR, with 5.87M parameters and 2.70G FLOPs.
- Adding FIFA to ResNet18 without pretraining lowers IV error from 8.63° to 7.93°, showing it transfers across backbones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FIFA to the ResNet18-based method (ablation) | ResNet18 gaze estimation · IV | mean angular error: 8.05◦ -> 7.29◦ | Helped |
| add FIFA to the ResNet18-based method (ablation) | ResNet18 gaze estimation · LBW | mean angular error: 6.41◦ -> 5.84◦ | Helped |
| add FIFA to the ResNet18 encoder (ablation) | ResNet18 without pretraining · IV | mean angular error: 8.63◦ -> 7.93◦ | Helped |
| add FIFA to the ResNet18 encoder (ablation) | ResNet18 without pretraining · LBW | mean angular error: 6.52◦ -> 6.30◦ | Helped |
| add FIFA to the ResNet50 encoder (ablation) | ResNet50 without pretraining · IV | mean angular error: 7.61◦ -> 7.42◦ | Helped |
| add FIFA to the ResNet50 encoder (ablation) | ResNet50 without pretraining · LBW | mean angular error: 6.40◦ -> 6.21◦ | Helped |
| add FIFA to the VGG-16 encoder (ablation) | VGG-16 without pretraining · IV | mean angular error: 8.77◦ -> 7.81◦ | Helped |
| add FIFA to the VGG-16 encoder (ablation) | VGG-16 without pretraining · LBW | mean angular error: 6.98◦ -> 6.69◦ | Helped |
| add FIFA to the ResNet18 encoder (ablation) | ImageNet-1k-pretrained ResNet18 · IV | mean angular error: 8.05◦ -> 7.65◦ | Helped |
| add FIFA to the ResNet18 encoder (ablation) | ImageNet-1k-pretrained ResNet18 · LBW | mean angular error: 6.41◦ -> 6.22◦ | Helped |
| add FIFA to the ResNet50 encoder (ablation) | ImageNet-1k-pretrained ResNet50 · IV | mean angular error: 7.37◦ -> 7.31◦ | Helped |
| add FIFA to the ResNet50 encoder (ablation) | ImageNet-1k-pretrained ResNet50 · LBW | mean angular error: 6.25◦ -> 6.10◦ | Helped |
Surprising findings
- Adding the encoder-only branch R alone degrades performance (10.61° on IV) relative to ResNet18, while adding FIFA on top of R recovers it.
Limitations
- Relies on adjacent frames; for IV, adjacent frames are synthesized by matching head poses or duplicating the image when no match exists.
- Authors note clustering in embeddings may be influenced by personalization, which may hurt gaze extraction.
- Improvements over prior methods on IV are small (e.g., 7.29° vs 7.33°).
Experimental setup
Datasets: LBW IV
Benchmarks: IV LBW
Metrics: mean angular error average precision at angle threshold FLOPs parameter count
Baselines: Gaze360 ResNet18 GazeTR XGaze GazePTR STAGE
Builds on: GazeTR ResNet18 STAGE GazePTR
Closest papers by meaning
- What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation (CVPR 2024 · similarity 0.89)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.86)
- From Feature to Gaze: A Generalizable Replacement of Linear Layer for Gaze Estimation (CVPR 2024 · similarity 0.85)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.85)
- OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild (NeurIPS 2025 · similarity 0.84)
- Unsupervised Gaze Representation Learning from Multi-view Face Images (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.