Research map / 3D vision / Face and shape
Unsupervised Gaze Representation Learning from Multi-view Face Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MV-DE, an unsupervised gaze representation learning framework using multi-view face images with a dual-encoder design and multi-view gaze representation swapping, robust to large head poses.
Problem
Unsupervised gaze representation learning from face images degrades under large head poses because head pose and gaze are coupled and eye-based methods fail when head rotation is large.
Approach
A Face Encoder (trained with eye masking, head pose estimation and reconstruction) is frozen, and a Gaze Encoder is trained to supply the missing eyeball-rotation information; multi-view consistency of eyeball rotation within a frame is enforced by swapping gaze representations across views during reconstruction.
Key results
- MV-DE averages 8.17° on ETH-XGaze, 6.73° on MPII-NV and 8.82° on EVE with 50-200 calibration samples, versus 15.16° for Cross-Encoder on ETH-XGaze.
- Multi-view swapping improves over the single-view dual-encoder ablation (DE) by up to 5.88° in the ETH-XGaze/MPII-NV settings.
- Used as unsupervised pretraining, MV-DE reduces cross-dataset error from 31.27° to 19.67° on ETH-XGaze→EVE (ETH-XGaze pretraining, EVE-to-ETH-XGaze baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the multi-view gaze-representation swapping constraint to the dual-encoder training (ablation) | MV-DE gaze representation learning · ETH-XGaze | gaze estimation error (°), 50/100/200-shot average: 20.34 -> 8.17 | Helped |
| add the multi-view gaze-representation swapping constraint to the dual-encoder training (ablation) | MV-DE gaze representation learning · MPII-NV | gaze estimation error (°), 50/100/200-shot average: 11.04 -> 6.73 | Helped |
| add the multi-view gaze-representation swapping constraint to the dual-encoder training (ablation) | MV-DE gaze representation learning · EVE | gaze estimation error (°), 50/100/200-shot average: 9.99 -> 8.82 | Hurt |
| use MV-DE instead of Cross-Encoder | unsupervised gaze representation learning · ETH-XGaze | gaze estimation error (°), 50/100/200-shot average: 15.16 -> 8.17 | Helped |
| use MV-DE instead of Cross-Encoder | unsupervised gaze representation learning · MPII-NV | gaze estimation error (°), 50/100/200-shot average: 11.53 -> 6.73 | Helped |
| use MV-DE instead of Cross-Encoder | unsupervised gaze representation learning · EVE | gaze estimation error (°), 50/100/200-shot average: 7.79 -> 8.82 | Hurt |
| add head pose as an input to the MLP (ablation) | MV-DE gaze estimation head (d=64) · ETH-XGaze | gaze estimation error (°), 200-shot: 8.43 -> 7.84 | Helped |
| add head pose as an input to the MLP (ablation) | MV-DE gaze estimation head (d=64) · MPII-NV | gaze estimation error (°), 200-shot: 7.44 -> 6.39 | Helped |
| add head pose as an input to the MLP (ablation) | MV-DE gaze estimation head (d=64) · EVE | gaze estimation error (°), 200-shot: 8.68 -> 8.67 | No clear effect |
| pretrain the gaze encoder with MV-DE on ETH-XGaze before labeled calibration | gaze estimation model · EVE test split | gaze estimation error (°): 31.27 -> 19.67 | Helped |
| pretrain the gaze encoder with MV-DE on MPII-NV before labeled calibration | gaze estimation model · EVE test split | gaze estimation error (°): 31.27 -> 25.42 | Helped |
| pretrain the gaze encoder with MV-DE on ETH-XGaze before labeled calibration | gaze estimation model · EyeDiap | gaze estimation error (°): 23.58 -> 10.4 | Helped |
Surprising findings
- Cross-Encoder performs better than MV-DE on EVE (frontal faces, limited head pose), where MV-DE is only comparable.
Limitations
- Gains are limited on frontal-face datasets such as EVE where head pose range is small.
- Requires multi-view captured data with frame-level synchronization for the swapping constraint.
- Needs a few-shot calibration set with gaze labels for the MLP head.
Experimental setup
Datasets: ETH-XGaze MPII-NV EVE EyeDiap
Benchmarks: ETH-XGaze MPII-NV EVE EyeDiap
Metrics: gaze estimation angular error (degrees)
Baselines: BYOL SimCLR Cross-Encoder DE (ours w/o MV)
Builds on: Cross-Encoder Masked Autoencoders
Closest papers by meaning
- UVAGaze: Unsupervised 1-to-2 Views Adaptation for Gaze Estimation (AAAI 2024 · similarity 0.87)
- Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels (CVPR 2025 · similarity 0.86)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.86)
- De^2Gaze: Deformable and Decoupled Representation Learning for 3D Gaze Estimation (CVPR 2025 · similarity 0.86)
- OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild (NeurIPS 2025 · similarity 0.85)
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.