Research map / Vision-language and multimodal / Multimodal LLMs

In-Context Learning Can Help Vision Language Models Overcome Training Prior

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that label-balanced in-context demonstrations let VLMs override training priors on prior-conflicting counterfactual images, with gains concentrated there and attention shifting toward grounding cues, but no gains for weak capabilities like high-count counting.

Problem

Vision-language models often fail on images that violate learned training regularities; it is unclear whether these failures reflect missing visual capability or a learned prior overriding available visual evidence.

Approach

Use controlled visual in-context learning with label-balanced demonstrations and prior-consistent vs. prior-conflicting counterfactual benchmarks to test whether demonstrations recover correct visually grounded answers, and analyze attention and token masking to attribute gains to reduced reliance on prior cues.

Key results

• Qwen3-VL 235B gains +27.2% on counterfactual complex examples at 16 shots while real accuracy changes +2.9%.
• Qwen3-VL 32B gains +15.1% on counterfactual examples, with real accuracy at -1.6%.
• Across five ~30B VLM families, 0-to-16-shot complex accuracy gains range from +9.7% (Qwen3-VL 32B) to +30.3% (Qwen3.5 27B).

Surprising findings

• Demonstrations do not reliably improve high-cardinality counting on synthetic shapes; some counts fall below zero-shot.
• Caption-only object direction accuracy falls below chance on counterfactual examples, consistent with configuration priors.

Limitations

• Demonstrations do not strengthen weak capabilities such as high-count counting.
• Complex-setting benchmark is small (1,160 examples) and counterfactuals are generated with Gemini editing.
• Model-specific training frequencies are unavailable, so prior strength is inferred indirectly.

Experimental setup

Datasets: VLMs are Biased (counting) WhatsUp COCO spatial GQA spatial EgoOrientBench

Benchmarks: counting color identification shape identification object direction spatial relation

Metrics: exact-match accuracy accuracy gain over 0-shot attention share on prior vs grounding cues

Baselines: zero-shot prompting Qwen2.5-VL Qwen3.5 InternVL3.5 Gemma-4

Builds on: logit lens LatentLens EgoOrientBench WhatsUp

Closest papers by meaning

• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.93)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.90)
• Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.90)
• Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain (ICML 2026 · similarity 0.90)
• VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.90)