Research map / 3D vision / Depth and stereo
De^2Gaze: Deformable and Decoupled Representation Learning for 3D Gaze Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes De2Gaze, a lightweight model-aware 3D gaze estimator that uses a learnable 3D eyeball model to project sparse edge points as references for deformable sparse attention and a dual-branch decoder to decouple invariant and variable eye features.
Problem
3D gaze estimation from near-eye infrared images suffers from sensitivity to local noise (light spots, blur) when using dense features, high computational cost, and entangled representations between eyeball models and gaze directions.
Approach
Combines a differentiable 3D eyeball model whose projected pupil/iris edge points serve as reference points for deformable sparse attention, with a spatial decoupling network that uses separate invariant (eyeball geometry) and variable (gaze, pupil, iris) decoder branches, jointly initialized query-by-query.
Key results
- On TEyeD subset A, gaze-only supervised De2Gaze reaches 0.54° 3D gaze error versus 1.04° for the previous best model-aware method (about 48% reduction).
- With gaze, semantic and center supervision, De2Gaze achieves 0.96° 3D gaze error and 93.4% IoU on subset A.
- On cross-subject subset B, De2Gaze achieves 2.61° 3D gaze error with gaze-only supervision, versus 3.03° for Nikola et al.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use De2Gaze instead of Nikola et al. | 3D gaze estimation · TEyeD-subset A | 3D gaze error (°): 1.04 -> 0.54 | Helped |
| Use De2Gaze instead of Nikola et al. | 3D gaze estimation · TEyeD-subset B | 3D gaze error (°): 3.03 -> 2.61 | Helped |
| Use De2Gaze with Sem. + Gaze + Cent. instead of Nikola et al. with Sem. | semantic segmentation · TEyeD-subset A | semantic IoU: 92.5% -> 93.4% | Helped |
| Use De2Gaze with Sem. + Gaze + Cent. instead of Nikola et al. with Sem. | semantic segmentation · TEyeD-subset B | semantic IoU: 86.6% -> 88.7% | Helped |
| Add dual-branch SDNet instead of using single-branch decoding (ablation) | De2Gaze · TEyeD-subset A | 3D gaze error (°): 1.18 -> 0.54 | Helped |
| Add dual-branch SDNet instead of using single-branch decoding (ablation) | De2Gaze · TEyeD-subset B | semantic IoU: 83.5% -> 88.7% | Helped |
| Add the learnable 3D eye model (ablation) | De2Gaze · TEyeD-subset A | 3D gaze error (°): 1.54 -> 0.54 | Helped |
| Add the learnable 3D eye model (ablation) | De2Gaze · TEyeD-subset B | 3D gaze error (°): 4.66 -> 2.61 | Helped |
| Add SPG with projected edge-point initialization instead of random query and reference-point initialization (ablation) | De2Gaze · TEyeD-subset A | 3D gaze error (°): 0.98 -> 0.54 | Hurt |
| Add SPG with projected edge-point initialization instead of random query and reference-point initialization (ablation) | De2Gaze · TEyeD-subset B | 3D gaze error (°): 2.98 -> 2.61 | Helped |
| Use DSA instead of standard cross-attention (ablation) | De2Gaze · TEyeD-subset A | 3D gaze error (°): 1.11 -> 0.54 | Helped |
| Use DSA instead of standard cross-attention (ablation) | De2Gaze · TEyeD-subset B | semantic IoU: 84.1% -> 88.7% | Helped |
Surprising findings
- Adding 2D semantic supervision alone yields poor 3D gaze accuracy (20.12°), indicating many 3D eyeball configurations share the same 2D semantics.
- Adding extra 2D eyeball-center and edge supervision slightly degraded 3D gaze accuracy in some settings, which the authors attribute to the unmodeled kappa angle.
Limitations
- Does not model the kappa angle between optical and visual axes, which limits 3D gaze accuracy.
- Adding 2D semantic and center supervision did not uniformly improve gaze accuracy.
- Cross-subject error is higher than within-subject error.
Experimental setup
Datasets: TEyeD
Benchmarks: TEyeD subset A TEyeD subset B
Metrics: 3D gaze angular error 2D gaze error semantic IoU 2D pupil/eye center error
Baselines: TEyeD NVGaze Transformer-based QueryDETR Nikola et al. model-aware 3D gaze
Builds on: deformable attention DETR3D model-aware 3D gaze estimation
Closest papers by meaning
- GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding (CVPR 2025 · similarity 0.89)
- Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels (CVPR 2025 · similarity 0.89)
- 3D Prior Is All You Need: Cross-Task Few-shot 2D Gaze Estimation (CVPR 2025 · similarity 0.87)
- OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild (NeurIPS 2025 · similarity 0.86)
- From Feature to Gaze: A Generalizable Replacement of Linear Layer for Gaze Estimation (CVPR 2024 · similarity 0.86)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.