Research map / 3D vision / 3D generation
GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GA3CE, a transformer-based 3D gaze estimator that takes 3D pose and SAM-derived object positions, normalizes them into an egocentric frame, and encodes them with D3 positional encoding to predict 3D gaze direction from RGBD input.
Problem
Estimating 3D gaze direction in unconstrained settings (distant or backward-facing subjects, no close-up eye view) where 2D appearance varies greatly with camera pose and prior methods use depth only in non-learnable post-processing.
Approach
Treat the subject's 3D pose and scene object positions as 3D context, align them to a head-centered egocentric frame via a cyclotorsion-based rotation, and encode direction and distance separately (D3 positional encoding) before a transformer decodes gaze residuals.
Key results
- On GFIE, 3D MAE of 11.1 vs 17.7 for GFIE (about 37% improvement).
- On GAFA single-frame setting, average 3D MAE improves about 13% over GAFA (19.9 vs 22.9).
- On CAD-120 generalization (trained on GFIE), 3D MAE 25.2 vs 27.3 for GFIEhead without GFM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate gaze with GA3CE instead of GFIE | 3D gaze estimation · GFIE | 3D MAE: 17.7 -> 11.1 | Helped |
| Add gaze-following modules | Ours · GFIE | 3D MAE: 11.1 -> 10.6 | Helped |
| Use the proposed method instead of GFIEhead | Models trained on GFIE · CAD-120 | 3D MAE: 27.3 -> 25.2 | Helped |
| Use Ours + GFM instead of GFIE | Models trained on GFIE · CAD-120 | 3D MAE: 19.8 -> 15.8 | Helped |
| Use the proposed method instead of GAFA | Single-frame gaze estimation · GAFA All | 3D MAE: 22.9 -> 19.9 | Helped |
| Add 3D pose (ablation) | Appearance-based model · GFIE | 3D MAE: 19.4 -> 13.1 | Helped |
| Add 3D pose (ablation) | Appearance-based model · GAFA | 3D MAE: 22.9 -> 20.3 | Helped |
| Add object positions to appearance and 3D pose (ablation) | Appearance + Pose · GFIE | 3D MAE: 13.1 -> 11.1 | Helped |
| Add object positions to appearance and 3D pose (ablation) | Appearance + Pose · GAFA | 3D MAE: 20.3 -> 19.9 | Helped |
| Add GA3CE to 3D context (ablation) | Appearance + 3D · GFIE | 3D MAE: 18.9 -> 11.1 | Helped |
| Add GA3CE to 3D context (ablation) | Appearance + 3D · GAFA | 3D MAE: 21.2 -> 19.9 | Helped |
| Add egocentric transformation (ablation) | GA3CE · GFIE | 3D MAE: 14.2 -> 11.1 | Helped |
Surprising findings
- Simply adding 3D pose and object positions without the GA3CE encoding gives only modest gains (e.g., 19.4 to 18.9 on GFIE for appearance+3D).
Limitations
- Object representation uses only 3D positions without category-agnostic semantic appearance features.
- Gains are smaller on GAFA where subjects interact less with objects.
- Relies on depth maps (real or zero-shot) and 2D bounding boxes for head and body.
Experimental setup
Datasets: GFIE CAD-120 GAFA
Benchmarks: GFIE CAD-120 GAFA
Metrics: 3D mean angular error AUC L2 distance 3D distance 2D MAE
Baselines: GFIE GAFA GAFAtemporal Rt-Gene Gaze360 GazeFollow Lian Chong XGaze
Builds on: GFIE GAFA SAM transformer (DETR-style encoder-decoder) sinusoidal positional encoding
Closest papers by meaning
- Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels (CVPR 2025 · similarity 0.89)
- De^2Gaze: Deformable and Decoupled Representation Learning for 3D Gaze Estimation (CVPR 2025 · similarity 0.89)
- 3D Prior Is All You Need: Cross-Task Few-shot 2D Gaze Estimation (CVPR 2025 · similarity 0.88)
- OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild (NeurIPS 2025 · similarity 0.88)
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.86)
- From Feature to Gaze: A Generalizable Replacement of Linear Layer for Gaze Estimation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.