Research map / Vision-language and multimodal / Multimodal LLMs

VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VisRes Bench, a ~19k-image, image-only four-choice benchmark across three levels (perceptual completion, single-attribute Raven-style rules, multi-attribute compositional rules) that shows state-of-the-art VLMs perform near chance on many subtasks.

Problem

VLMs may rely on linguistic priors rather than genuine visual reasoning; existing benchmarks rarely structure visual reasoning across perceptual, single-attribute, and multi-attribute levels in naturalistic imagery without textual cues.

Approach

A level-structured, image-only multiple-choice benchmark built on natural photos: Level 1 uses patch and global occlusion completion with perturbations and DINOv2-hard distractors, Level 2 uses single-attribute 3x3 Raven-style grids, and Level 3 uses multi-attribute coupled, independent, and spiral rules, plus diagnostic experiments (resolution, text verbalization, MAE retrieval, SFT).

Key results

• GPT-5 averages 31.10% on Level 1, 49.79% on Level 2, and 34.39% on Level 3 under guided prompting with thinking, against 25% chance.
• Human participants average about 91% across Level 1 tasks, versus roughly 24.5% for the original Qwen2.5-3B, which rises to 43.7% after SFT.
• GPT-5 reaches 85.0% on text-verbalized Level 2 versus 50.0% in the visual setting, indicating a visual-to-symbolic gap.

Surprising findings

• Several models score below chance (25%) because they loop in reasoning and fail to answer within the 32k context limit.
• Text-verbalized GPT-5 gets 85% on Level 2 while visual accuracy is ~50%, suggesting the bottleneck is visual feature extraction rather than reasoning.

Limitations

• Benchmark data and code release is not stated in the text.
• Level 2 and 3 labels for color and count rely partly on automated pipelines (Molmo, GPT-5) with manual checks on only a subset.
• Many open-source models are near chance without thinking mode; results depend on prompt and reasoning settings.

Experimental setup

Datasets: Google Street View scenes public web images Level-1 SFT training set

Benchmarks: VisRes Level 1 VisRes Level 2 VisRes Level 3

Metrics: accuracy

Baselines: GPT-4o Gemini-2.5 Qwen3-VL-30B Qwen3-VL-32B Kimi-VL-A3B MiMo-VL-7B-RL GLM-4.1V-9B InternVL3.5-8B

Builds on: Raven's Progressive Matrices DINOv2 MAE Molmo

Closest papers by meaning

• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.93)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.93)
• Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs (ICLR 2026 · similarity 0.92)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.91)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.91)
• Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.91)