Research map / Vision-language and multimodal / Human and embodied modeling

Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded visual concept learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Counterfactual Semantic Saliency (CSS), a black-box causal ablation metric, and uses it with a 16,289-response human psychophysics study to show VLMs diverge from humans in scene importance, mainly through over-reliance on object size.

Problem

Whether modern vision-language models prioritize objects in complex natural scenes the way humans do, given that white-box interpretability is impractical for closed-source VLMs and passive metrics cannot isolate causal feature importance.

Approach

Measure object importance by causally removing each object via generative inpainting (Nano Banana 2, SAM3 masks) and computing the semantic shift (1 minus mean cosine similarity of sentence embeddings) between factual and counterfactual VLM descriptions, then compare these rankings against human descriptions of the same stimuli.

Key results

• Human-human Top-1 consistency reached 73% while the 19 evaluated VLMs ranged from 57% to 65%.
• Mean Kendall's tau between humans was 0.58, versus 0.37 to 0.51 for models.
• Size bias correlation with the Top-1 accuracy gap was r = -0.6130 (permutation p = 0.0211), the strongest of the four hypothesized drivers.

Surprising findings

• Attention and Grad-CAM white-box maps correlate negatively with CSS (Kendall's tau about -0.24 to -0.34).
• Reasoning models did not achieve higher CSS alignment than non-reasoning counterparts.

Limitations

• Relies on generative inpainting being artifact-free, validated by 14 crowdsourced annotators rather than fully verified.
• Only 307 scenes from the OSIE dataset, with ablation limited to 3-6 objects per scene.
• Person bias classification and object lists depend on GPT-4o and GPT-5.4 judgments.

Experimental setup

Datasets: OSIE

Benchmarks: CSS counterfactual dataset

Metrics: Top-1 Accuracy Kendall's tau bias correlation (Spearman/point-biserial) bootstrap significance permutation test

Baselines: Grad-CAM Attention-Guided CAM (AGCAM) raw attention maps human-human consistency

Builds on: SAM3 Nano Banana 2 OSIE dataset GBVS Grad-CAM

Closest papers by meaning

• CF-VLM：CounterFactual Vision-Language Fine-tuning (NeurIPS 2025 · similarity 0.89)
• Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation (AAAI 2026 · similarity 0.88)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.88)
• Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
• When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding (NeurIPS 2025 · similarity 0.88)
• Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward (CVPR 2026 · similarity 0.88)