Research map / Vision-language and multimodal / CLIP and zero-shot
CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP-Gaze, a domain-generalization gaze estimator that uses frozen CLIP to build language-derived gaze-irrelevant features, pushes gaze features away from them, and adds personalized prompt tuning and a rank loss over sample pairs.
Problem
Appearance-based gaze estimators degrade substantially across domains (appearance, wearables, image quality), and existing domain generalization methods cover only a few gaze-irrelevant factors.
Approach
Use CLIP text encoder to build gaze-irrelevant features from flexible language descriptions of many factors, separate gaze-relevant features from them in CLIP's feature space, personalize prompts via 3DMM identity parameters (PCO), and re-rank pairwise feature similarities against label similarities.
Key results
- Under the DG setting, CLIP-Gaze achieves 6.97 average angular error versus 8.65 for the baseline and 7.63 for Xu et al.
- With 100 target samples fine-tuning (SDA), CLIP-Gaze reaches 5.07 average error versus 5.61 for the fine-tuned baseline.
- Ablation on prompt tuning: PCO gives 6.97 average, outperforming CoOp (7.36) and CoCoOp (7.81).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CLIP-Gaze instead of the strongest reported DG baseline | ResNet-18 gaze estimation · ETH-XGaze→MPIIFaceGaze | angular error (degrees): 6.50 -> 6.41 | Helped |
| apply CLIP-Gaze instead of the strongest reported DG baseline | ResNet-18 gaze estimation · ETH-XGaze→EyeDiap | angular error (degrees): 7.44 -> 7.51 | Hurt |
| apply CLIP-Gaze instead of the strongest reported DG baseline | ResNet-18 gaze estimation · Gaze360→MPIIFaceGaze | angular error (degrees): 7.03 -> 6.89 | Helped |
| apply CLIP-Gaze instead of the strongest reported DG baseline | ResNet-18 gaze estimation · Gaze360→EyeDiap | angular error (degrees): 7.27 -> 7.06 | Helped |
| add PCO and feature rank loss (ablation) | CLIP-Gaze− · Cross-domain average | angular error (degrees): 7.71 -> 6.97 | Helped |
| remove text prompt tuning (ablation) | CLIP-Gaze · Cross-domain average | angular error (degrees): 6.97 -> 7.40 | Hurt |
| use PCO instead of CoOp prompt tuning (ablation) | CLIP-Gaze · Cross-domain average | angular error (degrees): 7.36 -> 6.97 | Helped |
| use PCO instead of CoCoOp prompt tuning (ablation) | CLIP-Gaze · Cross-domain average | angular error (degrees): 7.81 -> 6.97 | Helped |
| use PCO instead of CoCoOp conditioned on one user face image (ablation) | CLIP-Gaze · Cross-domain average | angular error (degrees): 7.73 -> 6.97 | Helped |
| use appearance, wearable, and image quality factors together rather than appearance factors only (ablation) | CLIP-Gaze · Cross-domain average | angular error (degrees): 7.36 -> 6.97 | Helped |
| add CLIP feature alignment loss (ablation) | Baseline · Cross-domain average | angular error (degrees): 8.65 -> 8.38 | Helped |
| add gaze-irrelevant feature separation loss after CLIP alignment (ablation) | Model with CLIP feature alignment loss · Cross-domain average | angular error (degrees): 8.38 -> 7.37 | Helped |
Surprising findings
- Plain CLIP-Gaze without PCO and rank loss (CLIP-Gaze-) reaches only 7.71, while CoCoOp prompt tuning performs worse than CoOp, attributed to gaze-relevant information leaking into instance-conditional prompts.
Limitations
- Gaze-irrelevant features rely on hand-enumerated factor descriptions and pseudo-labels from CLIP for CelebA attributes.
- Performance on D_G to D_D is only similar to the best method rather than superior.
- Requires frontal face images per identity and 3DMM fitting for personalization.
Experimental setup
Datasets: ETH-XGaze Gaze360 MPIIFaceGaze EyeDiap CelebA
Benchmarks: MPIIFaceGaze EyeDiap
Metrics: angular error (degrees)
Baselines: Baseline (ResNet-18 with angular loss) PureGaze CDG Xu et al. PnP-GA RUDA CRGA LatentGaze UnReGA CoOp CoCoOp
Builds on: CLIP CoOp CoCoOp CRLoss PureGaze
Closest papers by meaning
- Test-Time Personalization with Meta Prompt for Gaze Estimation (AAAI 2024 · similarity 0.89)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.89)
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.88)
- From Feature to Gaze: A Generalizable Replacement of Linear Layer for Gaze Estimation (CVPR 2024 · similarity 0.87)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.87)
- Voila-A: Aligning Vision-Language Models with User's Gaze Attention (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.