Research map / Vision-language and multimodal / Multimodal LLMs
OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a LoRA-tuned Qwen3-VL-4B into a unified multi-person gaze model with a JSON language branch, a heatmap branch over hidden states, and head-embedding token injection, reaching SOTA on gaze following, semantic gaze, and social gaze benchmarks.
Problem
Gaze following models are limited to spatial localization, process people sequentially, and lack semantic/social reasoning; off-the-shelf VLMs localize poorly because coordinates are generated as discrete text tokens.
Approach
A dual-branch VLM decoder: an autoregressive JSON language branch for in/out status and category, plus a gaze branch that fuses per-person hidden-state tokens with image tokens to produce high-resolution heatmaps supervised by MSE; head crops are encoded by a Gaze360-pretrained ResNet-18 and injected into placeholder token slots for all persons in one forward pass.
Key results
- On GazeFollow, OmniGF reaches 0.091 Avg Dist. and 0.040 Min Dist., versus 0.099 for the best prior Gaze-LLE (ViT-L) and 0.104 for Gaze-LLE (ViT-B).
- On VSGaze social gaze, F1 LAEO improves from 0.590 (MTGS) to 0.742 and AP SA from 0.576 to 0.787.
- Zero-shot Qwen3-VL produces 0.340 Avg Dist. on GazeFollow, while LoRA-tuned language-only branch reaches 0.114 and the full model 0.091.
Surprising findings
- Zero-shot Qwen3-VL text-coordinate localization is very poor (0.340 Avg Dist.), showing off-the-shelf VLMs are unsuitable for gaze localization without the heatmap branch.
- Training only the heatmap branch without fine-tuning the language branch is worse (0.167) than the language-only LoRA model (0.114).
Limitations
- Substantial memory and compute footprint from the VLM backbone compared with CNN-based models.
- Frame-by-frame processing without temporal modeling on video datasets.
- Localization degrades slightly in very crowded scenes (over 15 individuals) due to attention dilution.
Experimental setup
Datasets: GazeFollow VideoAttentionTarget ChildPlay GazeHOI VSGaze Gaze360 VideoCoAtt UCO-LAEO
Benchmarks: GazeFollow VideoAttentionTarget ChildPlay GazeHOI VSGaze
Metrics: Avg Dist. Min Dist. Dist. AP Accuracy@1 MultiAccuracy@1 GazeAcc F1 AP_SA
Baselines: Sharingan MTGS Gaze-LLE (ViT-B) Gaze-LLE (ViT-L) GazeVLM SemGaze Gupta Tafasca Chong
Builds on: Qwen3-VL Gaze-LLE Sharingan MTGS
Closest papers by meaning
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.93)
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.90)
- Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.89)
- Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · similarity 0.88)
- OmniGaze: Reward-inspired Generalizable Gaze Estimation in the Wild (NeurIPS 2025 · similarity 0.88)
- ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.