Research map / Vision-language and multimodal / Human and embodied modeling

Attention Alignment Between Humans and Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Human-like visual cognition in VLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares attention maps of four in-house captioning models (CNN/ViT encoders × LSTM/Transformer decoders) plus Molmo 7B-D and Qwen3.5 9B against human eye-tracking heatmaps on two tasks, and adds an exploratory TRIBE-based neural encoding analysis.

Problem

Whether encoder and decoder architecture of vision-language models, particularly the top-down attention mechanism, shape how closely model spatial attention matches human fixations during description and social-attention captioning tasks.

Approach

Factorial 2x2 design isolating encoder (CNN vs ViT) and decoder (additive-attention LSTM vs multi-head Transformer) with training distribution held fixed, measuring alignment to fixation heatmaps via Spearman correlation normalized to a human noise ceiling, plus task differentiation, entropy, lesion-based neglect simulation, and synthetic neural encoding.

Key results

• LSTM decoders reached 80-87% of the human noise ceiling versus 40-59% for Transformer decoders; CNN-LSTM was most aligned overall (85-87%).
• Molmo 7B-D reached 76-82% of ceiling and Qwen3.5 9B 53-69%.
• CNN-Transformer had the highest whole-brain TRIBE-simulated R^2 on the Social task despite ranking third in fixation alignment.

Surprising findings

• Fixation alignment and synthetic neural predictive accuracy dissociate: CNN-Transformer predicted synthetic brain activity best despite lower fixation alignment.
• ViT-Transformer, the least aligned model, was the most resilient to simulated hemispatial-neglect ablation.

Limitations

• Head selection for Transformer decoders was done on Social-task fixations and then evaluated on the same task, possibly inflating scores.
• Neural analyses use TRIBE-simulated rather than measured brain responses.
• Molmo and Qwen were not fine-tuned on the tasks, unlike the trained models, confounding comparison.

Experimental setup

Datasets: COCO 2014 COCO 2017 validation (Karpathy splits) lab-collected eye-tracking fixations (n=49) lab-collected social captioning dataset (~3,400 pairs)

Benchmarks: EyeLink 1000 human fixation heatmaps (200 images, Social and Describe tasks)

Metrics: Spearman correlation normalized to human noise ceiling normalized Shannon entropy between-task spatial alignment caption cosine similarity whole-brain R^2 from banded ridge encoding variance partitioning

Baselines: Molmo 7B-D Qwen3.5 9B CNN-LSTM CNN-Transformer ViT-LSTM ViT-Transformer

Builds on: Bahdanau additive attention TRIBE v2 CLIP

Closest papers by meaning

• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.90)
• See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.88)
• No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.88)
• Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.88)
• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.88)
• Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.88)