Research map / Detection and segmentation / Open-set and few-shot detection
Learning from Observer Gaze: Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects the Interactive-Gaze (IG) fixation dataset over HOI scenes and proposes Interactive Attention (IA), a CLIP-based model for zero-shot interaction-oriented attention prediction (ZeroIA), whose maps (and real human gaze) are used as extra supervision to improve HOI detectors.
Problem
Existing attention prediction focuses on instances like humans and objects, leaving interaction-oriented visual attention unexplored, and interactions are open-ended so models must generalize to unseen interaction categories without training on them.
Approach
Activate CLIP with interaction-oriented prompts and tiny learnable positional/visual adapters that produce scene-adaptive knowledge prototypes for human, object and interaction, then progressively perceive instances and then interactions via gated attention blocks to decode an attention heatmap; the resulting attention is also used to supervise existing HOI detectors.
Key results
- IA w/ CLIP-L reaches CC 0.4488 in the ZeroIA setting, outperforming the third-best MLNet by 52.8% on CC.
- Aligning IA-generated attention raises MUREN to 68.0 mAP on VCOCO Scenario #1 (vs 66.9 baseline) and STIP to 67.2 (vs 66.0).
- On HICO-det, MUREN w/ IA-L improves rare-set mAP from 27.36 to 29.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IA CLIP-large instead of the strongest reported baseline | Interaction-oriented attention prediction · IG ZeroIA | CC: 0.2937 -> 0.4488 | Helped |
| Use IA CLIP-large instead of the strongest reported baseline | Interaction-oriented attention prediction · IG fully supervised | CC: 0.3195 -> 0.5106 | Helped |
| Add IA-generated attention supervision | MUREN · VCOCO Scenario #1 | mAP: 66.9 -> 68.0 | Helped |
| Add IA-generated attention supervision | UPT · VCOCO Scenario #2 | mAP: 65.5 -> 67.0 | Helped |
| Add IA-generated attention supervision | STIP · HICO-det Rare | mAP: 28.15 -> 28.90 | Helped |
| Add genuine human attention supervision | UPT · VCOCO Scenario #1 | mAP: 59.8 -> 60.7 | Helped |
| Add positional adapter (PA) (ablation) | IA CLIP-base · IG ZeroIA | CC: 0.3652 -> 0.4013 | Helped |
| Add visual adapter (VA) (ablation) | IA CLIP-base · IG ZeroIA | CC: 0.3838 -> 0.4013 | Helped |
| Add human-object cognitive block (HOCB) (ablation) | IA CLIP-base · IG ZeroIA | CC: 0.3498 -> 0.4013 | Helped |
| Add interaction cognitive block (ICB) (ablation) | IA CLIP-base · IG ZeroIA | CC: 0.3249 -> 0.4013 | Helped |
Surprising findings
- Pseudo-attention labels generated by IA gave larger HOI gains than the limited set of real human gaze data.
- Raw CLIP similarity maps show opposite visualization and noisy activation, lacking interpretability.
Limitations
- Real human attention data is limited (about 4,475 samples from VCOCO), and mouse clicks are used to simulate gaze fixations.
- Methods trained on IG struggle with complex multi-instance scenarios such as 'push' and with few training samples.
Experimental setup
Datasets: VCOCO HICO-det Interactive-Gaze (IG)
Benchmarks: Interactive-Gaze (IG) VCOCO HICO-det
Metrics: CC KLdiv SIM AUC mAP
Baselines: ITTI GBVS DeepGaze I DeepGaze IIE UMB MLNet ConvNext SSwin Transformer CLIP-ViT-B/16 CLIP-ViT-L/14
Builds on: CLIP MUREN UPT STIP
Closest papers by meaning
- Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · similarity 0.87)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.86)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.86)
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.86)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.85)
- Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.