Research map / Vision-language and multimodal / Multimodal LLMs

Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically benchmarks zero-shot and LoRA-fine-tuned VLMs (mainly Qwen3-VL-8B) on gaze following and social gaze prediction via a VQA formulation, comparing against specialized vision gaze models.

Problem

Gaze understanding (gaze following and social gaze prediction) has been studied with specialized vision models, but it is unclear how well general-purpose vision-language models understand gaze zero-shot or after fine-tuning.

Approach

Reformulates gaze following (in/out plus gaze point) and social gaze tasks (LAEO, SA, LAH) as JSON-output VQA, evaluates four prompting strategies (base, CoT, structured CoT, in-context CoT) and LoRA fine-tuning on constructed QA pairs, and compares directly against pure vision gaze models.

Key results

• Zero-shot Qwen3-VL-8B-Instruct with in-context CoT prompting reaches avg L2 0.215 on GazeFollow, well behind MTGS-DINO-static at 0.094.
• Fine-tuned Qwen3-VL-8B reduces GazeFollow avg L2 to 0.110 and VAT social F1_LAEO reaches 0.929, close to the specialized model.
• On social gaze prediction, a plain prompt without CoT performs best among zero-shot variants, and zero-shot VLMs approach specialized models on LAH/LAEO/SA more closely than on gaze following.

Surprising findings

• Chain-of-thought prompting helps gaze following but gives limited or negative gains for social gaze prediction.
• Smaller Qwen3-VL-4B fine-tuned model generalizes cross-dataset as well as or better than the 8B model.

Limitations

• Zero-shot VLMs remain far behind specialized gaze models on precise gaze point localization.
• Social gaze comparisons use a re-sampled balanced subset and a reduced 1K test subset for closed-source models.
• Output format failures are mapped to default values, which penalizes non-compliant models.

Experimental setup

Datasets: GazeFollow VideoAttentionTarget ChildPlay VSGaze annotations

Benchmarks: GazeFollow VideoAttentionTarget ChildPlay

Metrics: avg L2 min L2 angular error AP in/out F1 (LAH, LAEO, SA)

Baselines: MTGS-DINO-static Gaze-LLE Qwen3-VL-8B-Instruct Qwen3-VL-32B-Instruct InternVL3.5-38B Gemini-3.1-Pro-Preview ChatGPT-5.4-Thinking

Builds on: Gaze-LLE MTGS LLaMA-Factory

Closest papers by meaning

• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.93)
• Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding (NeurIPS 2025 · similarity 0.91)
• MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.90)
• Voila-A: Aligning Vision-Language Models with User's Gaze Attention (NeurIPS 2024 · similarity 0.90)
• Toward Semantic Gaze Target Detection (NeurIPS 2024 · similarity 0.89)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.89)