Research map / Detection and segmentation / CNN features
TransGOP: Transformer-Based Gaze Object Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TransGOP, an end-to-end gaze object prediction model combining a DINO Transformer detector with a Transformer-based gaze autoencoder, object-to-gaze cross-attention, and a Gaze Box loss, reaching SOTA on GOO-Synth and GOO-Real.
Problem
Gaze object prediction requires predicting both the location and category of the object a person is looking at; prior CNN-based GOP models (GaTector) have weak object localization for dense retail scenes and optimize detector and gaze branches separately.
Approach
Use a DETR-like Transformer detector (DINO) jointly with a Transformer gaze autoencoder whose decoder cross-attends to the detector's encoder memory (object-to-gaze cross-attention), and train end-to-end with a Gaze Box loss that maximizes heatmap energy inside the ground-truth gaze box.
Key results
- TransGOP reaches 92.8% mSoC on GOO-Synth vs 67.9% for GaTector.
- On GOO-Real without pre-training, TransGOP gets 82.6% mSoC vs 62.4% for GaTector.
- Object detection AP on GOO-Synth rises to 87.6 for TransGOP vs 83.7 for DINO and 56.8 for GaTector.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GaTector with TransGOP | GOP model · GOO-Synth | mSoC: 67.9 -> 92.8 | Helped |
| replace GaTector with TransGOP | GOP model · GOO-Real, no pre-train | mSoC: 62.4 -> 82.6 | Helped |
| replace GaTector with TransGOP | GOP model · GOO-Real, pre-trained on GOO-Synth | mSoC: 71.2 -> 89.0 | Helped |
| jointly optimize with gaze box loss (ablation) | DINO with CNN-based gaze regressor · GOO-Synth | AP: 83.7 -> 84.1 | Helped |
| add energy aggregation loss (ablation) | DINO with CNN-based gaze regressor · GOO-Synth | AP: 83.7 -> 81.2 | Hurt |
| add gaze autoencoder (ablation) | DINO with CNN-based gaze regressor · GOO-Synth | mSoC: 84.9 -> 89.8 | Helped |
| add gaze box loss (ablation) | DINO with CNN-based gaze regressor · GOO-Synth | mSoC: 84.9 -> 90.4 | Helped |
| add cross-adapter (ablation) | TransGOP gaze autoencoder · GOO-Synth | mSoC: 90.8 -> 92.8 | Helped |
| use object-to-gaze instead of gaze-to-object cross-attention (ablation) | TransGOP · GOO-Synth | mSoC: 79.4 -> 92.8 | Helped |
| replace GaTector with TransGOP | Gaze estimator · GOO-Synth | AUC: 0.957 -> 0.963 | Helped |
| replace GaTector with TransGOP | Gaze estimator · GOO-Synth | Dist.: 0.073 -> 0.079 | Hurt |
| replace GaTector with TransGOP | Gaze estimator · GOO-Synth | Ang.: 14.9 -> 13.3 | Helped |
Surprising findings
- The energy aggregation loss from GaTector lowers object detection AP (81.2 vs 83.7) when added to the DINO baseline.
Limitations
- L2 distance error slightly worse than GaTector on GOO-Synth (0.079 vs 0.073) due to gaze box loss trade-off.
- Object detection performance on GOO-Real without pretraining is only compared in limited settings; mSoC95 remains low on GOO-Real (20.1 AP95).
Experimental setup
Datasets: GOO-Synth GOO-Real
Benchmarks: GOO-Synth GOO-Real
Metrics: mSoC mSoC50 mSoC75 mSoC95 AP AUC L2 distance angular error
Baselines: GaTector YOLOv4 YOLOv7 Deformable DETR DINO Recasens Lian Chong
Builds on: DINO GaTector energy aggregation loss DETR
Closest papers by meaning
- Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (CVPR 2025 · similarity 0.87)
- Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels (CVPR 2025 · similarity 0.86)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.86)
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.86)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.85)
- Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.