Research map / 3D vision / Human pose estimation
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gaze-LLE streamlines gaze target estimation by training a small 3-layer transformer decoder with head-position prompting on top of a frozen DINOv2 encoder, achieving SOTA on GazeFollow, VideoAttentionTarget, and ChildPlay.
Problem
Gaze target estimation (predicting where a person is looking in a scene) has relied on complex multi-branch architectures with separate head/scene encoders and auxiliary depth/pose models, which are hard to train and do not benefit from foundation model advances.
Approach
Use a single frozen DINOv2 scene feature map, add a learned head-position embedding to tokens at the person's head location (late in the pipeline), and decode gaze with a small transformer plus conv heatmap decoder, eliminating head branches, fusion modules, and multi-task losses.
Key results
- Gaze-LLE ViT-L reaches 0.958 AUC and 0.099 Avg L2 on GazeFollow, state of the art over prior methods such as Tafasca et al. (0.944 AUC).
- With only ~2.8M learnable parameters (ViT-B), Gaze-LLE matches or beats prior methods with 25M-135M+ parameters across GazeFollow and VideoAttentionTarget.
- Gaze-LLE achieves SOTA on ChildPlay (ViT-L: 0.951 AUC, 0.101 L2) and reaches SOTA in under 1.5 GPU-hours on an RTX4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gaze-LLE with a frozen DINOv2 ViT-L backbone instead of the strongest reported baseline | Gaze target estimation · GazeFollow test | AUC: 0.944 -> 0.958 | Helped |
| Use Gaze-LLE with a frozen DINOv2 ViT-L backbone instead of the strongest reported baseline | Gaze target estimation · VideoAttentionTarget | AUC: 0.917 -> 0.937 | Helped |
| Use Gaze-LLE with a frozen DINOv2 ViT-L backbone instead of the strongest reported baseline | Gaze target estimation · ChildPlay | AUC: 0.935 -> 0.951 | Helped |
| Add head prompting (ablation) | Gaze-LLE · GazeFollow test | AUC: 0.926 -> 0.956 | Helped |
| Replace DINOv2 with CLIP as the pretrained backbone (ablation) | Gaze-LLE · GazeFollow test | AUC: 0.958 -> 0.953 | Hurt |
| Replace DINOv2 with MAE as the pretrained backbone (ablation) | Gaze-LLE · GazeFollow test | AUC: 0.958 -> 0.947 | Hurt |
| Replace DINOv2 with a supervised ImageNet-1k backbone (ablation) | Gaze-LLE · GazeFollow test | AUC: 0.958 -> 0.928 | Hurt |
| Move head prompting from Layer 1 to Layer 3 (ablation) | Gaze-LLE · GazeFollow test | AUC: 0.956 -> 0.955 | No clear effect |
| Use a transformer decoder instead of a convolutional decoder (ablation) | Frozen DINOv2 gaze baseline with late head integration and H+S branches · GazeFollow | AUC: 0.932 -> 0.954 | Helped |
| Use a transformer decoder instead of a convolutional decoder (ablation) | Frozen DINOv2 gaze baseline with late head integration and a scene-only branch · GazeFollow | AUC: 0.916 -> 0.953 | Helped |
| Remove the separate head branch (ablation) | Frozen DINOv2 gaze baseline with a transformer decoder · GazeFollow | AUC: 0.954 -> 0.953 | No clear effect |
Surprising findings
- Naively substituting DINOv2 into prior multi-branch gaze architectures gives worse performance than the original backbones.
- Without a head prompt the model still predicts a gaze target for at least one person, suggesting scene features implicitly detect heads.
Limitations
- Performance is tied to the frozen encoder's quality; requires a large encoder which may be challenging for embedded systems.
- Largest performance drop on GOO-Real due to domain gap and annotation scheme differences.
Experimental setup
Datasets: GazeFollow VideoAttentionTarget ChildPlay GOO-Real
Benchmarks: GazeFollow VideoAttentionTarget ChildPlay GOO-Real
Metrics: AUC Avg L2 Min L2 AP (in/out) P.Head
Baselines: Chong et al. Miao et al. Gupta et al. Tafasca et al. Jin et al. Fang et al. Tonini et al.
Builds on: DINOv2 Recasens et al. GazeFollow two-branch architecture Tafasca et al. transformer gaze backbone
Closest papers by meaning
- CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model (AAAI 2024 · similarity 0.88)
- TransGOP: Transformer-Based Gaze Object Prediction (AAAI 2024 · similarity 0.87)
- Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels (CVPR 2025 · similarity 0.87)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.87)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.87)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.