Research map / Vision-language and multimodal / Human and embodied modeling
Voila-A: Aligning Vision-Language Models with User's Gaze Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Voila-A, an Otter-based VLM that injects gaze heatmaps via a Voila Perceiver Resampler, trained on GPT-4-annotated trace-aligned QA data (VOILA-COCO) and evaluated on a new real gaze test set (VOILA-GAZE).
Problem
Existing vision-language models do not align their focus with a human user's gaze attention in complex multi-object scenes, and gaze data is costly to collect for training.
Approach
Uses mouse-trace data from Localized Narratives, downsampled to resemble gaze, as a proxy for gaze; encodes gaze heatmaps into a Perceiver-style resampler with an added gaze term in attention, while freezing pretrained weights and unfreezing in stages to preserve pretrained knowledge.
Key results
- Voila achieves 0.41 win rate over Otter-base on the ablation with reward score -0.79 vs Otter-base -1.91 (coreference queries).
- Voila with in-context prompt and direct query reaches 0.77 win rate over Otter-base and reward 0.20.
- VOILA outperforms Otter and Kosmos-2 on both VOILA-COCO test and the real-gaze VOILA-GAZE set according to GPT-4 ranking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with gaze-aligned data and the Voila-A method | Voila-A · VOILA-COCO-Testset | GPT-4 ranking | Helped |
| train with gaze-aligned data and the Voila-A method | Voila-A · VOILA-GAZE | GPT-4 ranking | Helped |
| add in-context QA examples before the current query (ablation) | Voila · VOILA-COCO-Testset, direct and coreference queries | Response quality: 0.41 -> 0.46; -0.79 -> -0.02; 0.62 -> 0.77; 0.14 -> 0.20 | Helped |
| represent gaze as discrete position tokens instead of a heatmap (ablation) | Voila gaze integration · VOILA-COCO-Testset | Winning rate over Otter-base: 0.41 -> 0.19 | Hurt |
| represent gaze bounding box as an image patch instead of a heatmap (ablation) | Voila gaze integration · VOILA-COCO-Testset | Winning rate over Otter-base: 0.41 -> 0.36 | Hurt |
| represent gaze bounding box as discrete position tokens instead of a heatmap (ablation) | Voila gaze integration · VOILA-COCO-Testset | Winning rate over Otter-base: 0.41 -> 0.21 | Hurt |
| gradually fine-tune gaze-related weights before perceiver and cross-attention layers (ablation) | Voila · VOILA-COCO-Testset | Winning rate over Otter-base: 0.25 -> 0.41 | Helped |
| replace gradual unfreezing with fine-tuning gaze weights only (ablation) | Voila · VOILA-COCO-Testset | Winning rate over Otter-base: 0.41 -> 0.24 | Hurt |
| replace gradual unfreezing with gaze weights plus LoRA (ablation) | Voila · VOILA-COCO-Testset | Winning rate over Otter-base: 0.41 -> 0.23 | Hurt |
Surprising findings
- Gaze as discrete position tokens or bounding-box tokens underperformed Otter-base, while heatmap conditioning worked better.
- LoRA fine-tuning of the whole model performed worse than the staged unfreezing scheme.
Limitations
- Occasional hallucinations related to image content.
- Small training set and VOILA-GAZE test set of only 100 samples.
- Relies on GPT-4 generated annotations and GPT-4 judging, with sequence-order bias.
Experimental setup
Datasets: LN-COCO (Localized Narratives COCO subset) COCO captions VOILA-COCO VOILA-GAZE
Benchmarks: VOILA-COCO test set VOILA-GAZE
Metrics: GPT-4 ranking win rate reward model score
Baselines: Otter-base Kosmos-2
Builds on: Otter OpenFlamingo Perceiver Localized Narratives
Closest papers by meaning
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.90)
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.86)
- CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model (AAAI 2024 · similarity 0.86)
- OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.86)
- Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding (NeurIPS 2025 · similarity 0.86)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.