Research map / Label-efficient and robust learning / Semi-supervised learning
OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Semi-supervised 3D gaze estimation framework (OmniGaze) that pseudo-labels 1.4M unlabeled face images and uses a multimodal reward model (CLIP visual features, MLLM-generated gaze descriptions, 3D gaze vectors) to filter and reweight pseudo labels.
Problem
3D gaze estimation generalizes poorly across domains because labeled gaze data is scarce and not diverse; standard threshold-based pseudo-labeling does not apply to continuous regression outputs.
Approach
A reward model that scores the reliability of continuous gaze pseudo labels by jointly reasoning over CLIP visual embeddings, MLLM-generated scene-specific gaze descriptions encoded with CLIP text encoder, and the 3D direction vector of the pseudo label, plus student-teacher agreement; the scores both filter (threshold 0.5) and reweight the student's unlabeled loss, with periodic teacher refresh.
Key results
- In-domain: 2.97 on MPIIFaceGaze, 4.07 on EyeDiap, 5.40 on RT-Gene, 9.12 on Gaze360, 6.72 on IVGaze, beating prior SOTA (e.g., AGE-Net 3.61 on MPIIFaceGaze).
- Cross-domain: ETH-XGaze to MPIIFaceGaze 5.07 vs. AGG 5.91 and LG-Gaze 6.45 in the compared table.
- Zero-shot with ViT-B on unseen datasets: 3.44 (MPIIFaceGaze), 4.31 (EyeDiap), 9.09 (RT-Gene), versus 6.09 for FaRL-B on MPIIFaceGaze.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniGaze instead of the strongest listed baseline | In-domain gaze estimation · MPIIFaceGaze | angular error: 3.61 -> 2.97 | Helped |
| Use OmniGaze instead of the strongest listed baseline | Cross-domain gaze estimation · ETH-XGaze to EyeDiap | angular error: 6.75 -> 4.84 | Helped |
| Add large-scale unlabeled data to labeled training (ablation) | OmniGaze · MPIIFaceGaze zero-shot | angular error: 6.17 -> 4.97 | Helped |
| Add reward-based pseudo-label selection (ablation) | OmniGaze trained with labeled and unlabeled data · MPIIFaceGaze zero-shot | angular error: 4.97 -> 3.44 | Helped |
| Add scene-specific gaze descriptions and 3D direction-vector labels to the evaluator (ablation) | OmniGaze reward model · MPIIFaceGaze zero-shot | angular error: 4.52 -> 3.44 | Helped |
| Add scene-specific gaze descriptions to the evaluator (ablation) | OmniGaze reward model · MPIIFaceGaze zero-shot | angular error: 4.52 -> 3.69 | Helped |
| Add 3D direction-vector labels to the evaluator (ablation) | OmniGaze reward model · MPIIFaceGaze zero-shot | angular error: 4.52 -> 4.03 | Helped |
| Add confidence-based pseudo-label filtering (ablation) | OmniGaze · MPIIFaceGaze zero-shot | angular error: 4.97 -> 4.18 | Helped |
| Add confidence-based pseudo-label reweighting (ablation) | OmniGaze · MPIIFaceGaze zero-shot | angular error: 4.97 -> 3.84 | Helped |
| Add periodic pseudo-label updates (ablation) | OmniGaze · MPIIFaceGaze zero-shot | angular error: 3.77 -> 3.44 | Helped |
| Use online rather than pre-trained reward-model training (ablation) | OmniGaze · EyeDiap zero-shot | angular error: 5.03 -> 4.31 | Helped |
| Include student-prediction similarity in final confidence scores (ablation) | OmniGaze reward model · Zero-shot datasets | angular error | Helped |
Surprising findings
- Scene-specific MLLM gaze descriptions alone give only modest gains (e.g., 0.42 degrees on MPIIFaceGaze), so the main gains come from the combined semi-supervised strategy rather than the descriptions alone.
- Pre-training the reward model on labeled and unlabeled data performed worse (3.71 on MPIIFaceGaze) than training it online alongside the student (3.44).
Limitations
- Requires a large unlabeled dataset and repeated teacher/reward model refreshes, adding training cost.
- Reward model relies on VLMs and MLLMs (CLIP, InstructBLIP), which may carry their pretraining biases.
- Reports results for only a subset of configurations; the code and pre-trained models are promised but not yet released.
Experimental setup
Datasets: ETH-XGaze Gaze360 MPIIFaceGaze EyeDiap RT-Gene IVGaze CelebA VGGFace2 FaceSynthetics SFHQ-T2I VFHQ WebFace
Benchmarks: MPIIFaceGaze EyeDiap RT-Gene Gaze360 IVGaze ETH-XGaze
Metrics: angular error (degrees)
Baselines: FullFace RCNN Gaze360 RT-Gene XGaze CANet GazeTR AGE-Net 3DGazeNet PureGaze RAT Gaze-Consistent
Builds on: FixMatch FlexMatch SemiReward CLIP InstructBLIP
Closest papers by meaning
- Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels (CVPR 2025 · similarity 0.90)
- OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.88)
- GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding (CVPR 2025 · similarity 0.88)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.87)
- UVAGaze: Unsupervised 1-to-2 Views Adaptation for Gaze Estimation (AAAI 2024 · similarity 0.87)
- Suppressing Uncertainty in Gaze Estimation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.