Research map / Vision-language and multimodal / Multimodal LLMs
VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a procedurally generated benchmark suite testing nonlocal visual reasoning (comparative perception, saccadic search, smooth visual search) and finds flagship VLMs near random on several variants.
Problem
Existing VLM benchmarks show high scores on complex tasks but do not test whether models can chain evidence across distant image regions or perform basic primitive visual algorithms like comparison, sequential search, and contour tracing.
Approach
Three minimal-context synthetic tasks (Object Re-Identification, Visual Scavenger Hunt, Circuit Connections), each with variants that separate perception from natural-language shortcuts, mapped to human vision theories (Gestalt, FIT, visual routines) to evaluate nonlocal visual reasoning in VLMs.
Key results
- On Object Re-Identification Standard variant, no model significantly outperforms random chance (50%), while GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4 score 12-24 points higher on Unconnected and Pixel-Perfect variants but remain over 20 points below humans.
- On Visual Scavenger Hunt, only GPT-5, o4-mini, and Gemini 2.5 Pro significantly beat the ~9% random baseline; o4-mini's decomposed single-step accuracy was about 94% per step versus lower multi-step performance.
- On Circuit Connections, the best model Gemini 2.5 Pro reached 27% on Single Color variant, with all models well below the 99.5% human baseline and performance strongly tied to wire distance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Unconnected variant instead of Standard (ablation) | GPT-5 · Object Re-Identification | accuracy: 60% -> 77% | Helped |
| Use the Pixel-Perfect variant instead of Standard (ablation) | GPT-5 · Object Re-Identification | accuracy: 60% -> 71% | Helped |
| Use the Unconnected variant instead of Standard (ablation) | Gemini 2.5 Pro · Object Re-Identification | accuracy | Helped |
| Use the Pixel-Perfect variant instead of Standard (ablation) | Gemini 2.5 Pro · Object Re-Identification | accuracy | Helped |
| Use the Unconnected variant instead of Standard (ablation) | Claude Sonnet 4 · Object Re-Identification | accuracy | Helped |
| Use the Pixel-Perfect variant instead of Standard (ablation) | Claude Sonnet 4 · Object Re-Identification | accuracy | Helped |
| Guide the search with sequential single-step queries (ablation) | o4-mini · Visual Scavenger Hunt, 3-step sequence | accuracy: 44.5% -> 90.67% | Helped |
| Use Unique Colors instead of Single Color (ablation) | Gemini 2.5 Pro · Circuit Connections | accuracy: 27% -> 48% | Helped |
| Increase the Visual Scavenger Hunt chain length (ablation) | o4-mini · Visual Scavenger Hunt | accuracy | Hurt |
Surprising findings
- Models that fail the Standard Object Re-Identification variant perform much better on Unconnected and Pixel-Perfect variants, suggesting selective comparison rather than general comparative perception.
- o4-mini achieves roughly 90% single-step accuracy in decomposed queries but only ~44.5% on the full multi-step task, indicating the bottleneck is chaining steps rather than perception.
Limitations
- Synthetic setting does not capture the complexity of natural images.
- Evaluated on only 200 or 125 examples per variant for cost reasons.
- Human baseline based on the authors' self-evaluation of a limited number of examples.
Experimental setup
Benchmarks: Object Re-Identification Visual Scavenger Hunt Circuit Connections
Metrics: accuracy F1 (positive) F1 (negative) log-odds coefficients
Baselines: Human baseline Random guessing o3 o4-mini Claude 3.7 Sonnet
Builds on: BlindTest Ullman visual routines Feature Integration Theory
Closest papers by meaning
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
- Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.92)
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.91)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.91)
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.91)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.