Research map / Vision-language and multimodal / Human and embodied modeling
What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IVGaze, a vision-based in-vehicle gaze dataset from 125 subjects with a transparent-chessboard gaze target calibration, and GazeDPTR, a dual-stream gaze pyramid transformer that also extends to gaze zone classification.
Problem
In-vehicle driver gaze estimation is limited by the lack of annotated real-driving datasets, since gaze targets lie outside the driver-monitoring camera's field of view and conventional collection setups are impractical in vehicles.
Approach
A vision-based collection system uses an auxiliary depth camera and a transparent chessboard to calibrate out-of-FoV gaze targets; GazeDPTR fuses multi-level CNN features via a transformer and a normalized (perspective-rotated virtual camera) image stream with camera-pose positional encoding; gaze zone classification adds tri-plane intersection positional features.
Key results
- GazeDPTR achieves 6.71° angular error on IVGaze, versus 7.06° for XGaze and 7.33° for GazeTR.
- Using positional features from tri-plane projection plus visual features improves gaze zone classification AP to 81.8%, a 2.4% gain over visual features alone (79.4%).
- Sunglasses degrade performance sharply, with GazeDPTR at 16.41° versus about 6.6° without sunglasses-related accessories cases for glasses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| integrate multilevel features with a transformer | GazePTR · IVGaze | angular error: 7.33° -> 7.04° | Helped |
| use normalized and original images in a dual stream | GazeDPTR · IVGaze | angular error: 7.04° -> 6.71° | Helped |
| add the 1st-level feature to the 4th-level feature (ablation) | GazePTR · IVGaze | angular error: 7.35° -> 7.04° | Helped |
| add the 2nd-level feature to the 4th-level feature (ablation) | GazePTR · IVGaze | angular error: 7.35° -> 7.04° | Helped |
| add the 3rd-level feature to the 4th-level feature (ablation) | GazePTR · IVGaze | angular error: 7.35° -> 7.04° | Helped |
| replace the 4th-level feature with the 1st-level feature (ablation) | GazePTR · IVGaze | angular error: 7.35° -> 10.33° | Hurt |
| replace the 4th-level feature with the 2nd-level feature (ablation) | GazePTR · IVGaze | angular error: 7.35° -> 8.59° | Hurt |
| replace the 4th-level feature with the 3rd-level feature (ablation) | GazePTR · IVGaze | angular error: 7.35° -> 7.61° | Hurt |
| use original images instead of normalized images (ablation) | GazePTR · IVGaze | angular error: 7.04° -> 7.44° | Hurt |
| add original images to normalized images (ablation) | GazeDPTR · IVGaze | angular error: 7.04° -> 6.71° | Helped |
| remove camera-pose positional encoding (ablation) | GazeDPTR · IVGaze | angular error: 6.71° -> 7.03° | Hurt |
| replace normalization method [38] with the proposed normalization (ablation) | GazePTR · IVGaze | angular error: 8.53° -> 7.04° | Helped |
Surprising findings
- Deeper backbone XGaze matches GazePTR in angular error but is markedly worse at AP<2°, attributed to ResNet50 overlooking the small eye region.
- Removing camera-pose positional encoding from GazeDPTR yields performance comparable to using only normalized images (7.03° vs 7.04°).
Limitations
- Sunglasses cause large performance drops (over 15° error for most methods).
- Masks and extreme head poses complicate feature extraction; low-resolution in-vehicle face images.
- Gaze zone classification with positional features alone performs worse than visual features (75.3% vs 79.4%).
Experimental setup
Datasets: IVGaze
Benchmarks: IVGaze
Metrics: angular error average precision (AP) at <2°, <4°, <6°, <8° gaze zone classification average precision
Baselines: FullFace FullFace+ DWG Gaze360 XGaze GazeTR
Builds on: GazeTR ETH-XGaze FullFace Gaze360 data normalization
Closest papers by meaning
- FIFA: Fine-grained Inter-frame Attention for Driver's Video Gaze Estimation (CVPR 2025 · similarity 0.89)
- Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction (ICLR 2025 · similarity 0.84)
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.84)
- UrbanIng-V2X: A Large-Scale Multi-Vehicle, Multi-Infrastructure Dataset Across Multiple Intersections for Cooperative Perception (NeurIPS 2025 · similarity 0.83)
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.83)
- UVAGaze: Unsupervised 1-to-2 Views Adaptation for Gaze Estimation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.