Research map / 3D vision / 3D generation
Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST-WSGE, a two-stage self-training framework that turns 2D gaze-following labels (GazeFollow) into 3D pseudo-labels via geometric projection, and GaT, a modality-agnostic Swin3D-based transformer trained on both image and video gaze data, for in-the-wild 3D gaze estimation.
Problem
3D gaze estimation in unconstrained real-world settings is hard because in-the-wild 3D gaze datasets are scarce and lack diversity in appearance, head pose, and gaze distribution, and video and image data are hard to use jointly.
Approach
Use a model trained on 3D gaze data to supply the depth component for 2D gaze-following labels, rotating the predicted 3D gaze about the z-axis so its image projection matches the 2D ground truth; retrain on the resulting pseudo-3D labels. A single modality-agnostic transformer (GaT) learns from both image and video datasets by treating images as duplicated clips.
Key results
- ST-WSGE (GaT, Gaze360+GF) reaches 12.17° video and 13.19° image angular error on Gaze360 Full, beating the best prior video result (MSA+Seq 12.5°).
- On GFIE image, ST-WSGE with GazeFollow reduces error from 30.57° (supervised) to 21.48°, a roughly 30% relative gain (paper reports 38% vs supervised in its text).
- Training GaT with GFIE+MPS+GF gives cross-dataset Gaze360 Full image error of 21.59° vs 25.75° supervised.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GazeFollow with pseudo-3D gaze labels using ST-WSGE to Gaze360 training (ablation) | GaT trained on Gaze360 · Gaze360 Full | angular error (°): 13.64 -> 13.19 | Helped |
| Add GazeFollow with pseudo-3D gaze labels using ST-WSGE to Gaze360 training (ablation) | GaT trained on Gaze360 · Gaze360 Full video | angular error (°): 12.60 -> 12.17 | Helped |
| Add GazeFollow using weak 2D supervision to Gaze360 training (ablation) | GaT trained on Gaze360 · Gaze360 Full | angular error (°): 13.6 -> 13.1 | Helped |
| Use self-training with predicted 3D gaze labels rather than pseudo-3D labels aligned to the 2D target (ablation) | GaT with Gaze360 and GazeFollow · Gaze360 / GFIE / EDIAP / MPII | angular error (°) | Mixed |
| Add GazeFollow with ST-WSGE to GFIE training (ablation) | GaT trained on GFIE · Gaze360 Full | angular error (°): 30.57 -> 21.48 | Helped |
| Add GazeFollow with ST-WSGE to GFIE training (ablation) | GaT trained on GFIE · GFIE | angular error (°): 15.70 -> 12.99 | Helped |
| Train GaT on both image and video data instead of image data alone (ablation) | GaT trained on Gaze360 · Gaze360 Full video | angular error (°): 13.82 -> 12.60 | Helped |
| Train GaT on both image and video data instead of video data alone (ablation) | GaT trained on Gaze360 · Gaze360 Full video | angular error (°): 12.31 -> 12.60 | Helped |
| Use GaT instead of Swin(2D)-LSTM (ablation) | Image-and-video gaze estimation · Gaze360 Full video | angular error (°): 13.02 -> 12.60 | Helped |
| Use GaT instead of Swin(2D)-Tr (ablation) | Image-and-video gaze estimation · Gaze360 Full video | angular error (°): 12.91 -> 12.60 | Helped |
| Use both image and video modalities instead of video-only training (ablation) | ST-WSGE (GaT) trained on Gaze360 and GazeFollow · Gaze360 Full video | angular error (°): 12.1 -> 12.2 | Hurt |
| Use both image and video modalities instead of video-only training (ablation) | ST-WSGE (GaT) trained on GFIE and GazeFollow · Gaze360 Full video | angular error (°): 24.2 -> 21.1 | Helped |
Surprising findings
- Adding image-only training data (GazeFollow treated as images) improved video gaze generalization.
- Pure self-training using only the model's predicted 3D gaze (no geometric projection to 2D labels) lacked gaze-distribution diversity and underperformed ST-WSGE.
Limitations
- Generalization gains are bounded by the diversity of the 3D pre-training datasets; GFIE-trained models improve less on frontal G360 40.
- Pseudo-3D depth relies on the prior z-estimate of a model trained on unconstrained 3D datasets.
- Gains on frontal/constrained benchmarks (MPIIFaceGaze, EYEDIAP) are mixed relative to weakly-supervised 2D-only baselines.
Experimental setup
Datasets: Gaze360 GFIE MPSGaze GazeFollow MPIIFaceGaze EYEDIAP
Benchmarks: Gaze360 Full Gaze360 180 Gaze360 40 Gaze360 Face Gaze360 Back GFIE MPIIFaceGaze EYEDIAP
Metrics: angular error (degrees)
Baselines: Gaze360 Kothari et al. MSA+Seq MCGaze 3DGazeNet L2CS GazeTR SAM-LSTM
Builds on: Swin Transformer Kothari et al. LAEO weakly-supervised gaze Gaze360
Closest papers by meaning
- OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild (NeurIPS 2025 · similarity 0.90)
- GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding (CVPR 2025 · similarity 0.89)
- De^2Gaze: Deformable and Decoupled Representation Learning for 3D Gaze Estimation (CVPR 2025 · similarity 0.89)
- 3D Prior Is All You Need: Cross-Task Few-shot 2D Gaze Estimation (CVPR 2025 · similarity 0.88)
- Bootstrap AutoEncoders With Contrastive Paradigm for Self-supervised Gaze Estimation (ICML 2024 · similarity 0.88)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.