Research map / Vision-language and multimodal / Multimodal LLMs

VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VLM-SubtleBench, a 13K-item benchmark of subtle comparative reasoning over image pairs across ten difference types and six domains, and evaluates proprietary and open-source VLMs against humans, with prompting, synthetic controlled tests, and fine-tuning analyses.

Problem

Existing comparative reasoning benchmarks for VLMs use image pairs with large, salient differences in mostly natural images, so they do not measure fine-grained subtle comparison needed in industrial, medical, and aerial settings.

Approach

A benchmark of nearly identical image pairs spanning ten difference types (Attribute, State, Emotion, Temporal, Spatial, Existence, Quantity, Quality, Viewpoint, Action) and six domains, built from existing annotated datasets plus model-assisted editing, with human-written difference captions, plus controlled synthetic experiments that vary difficulty factors.

Key results

• GPT-5-thinking achieves the highest average accuracy of 77.8% on VLM-SubtleBench MCQ versus 95.5% human evaluation, with gaps over 30 points on spatial, temporal, and viewpoint reasoning.
• Fine-tuning Qwen2.5-VL-7B on the validation split raises its average accuracy from 59.4% to 69.5%.
• Concatenating two images into one input degrades accuracy in 9 of 10 categories, while reasoning prompts improve GPT-5-main from 71.3% to 73.1%.

Surprising findings

• Two-step describe-then-answer prompting slightly lowers performance because the model often describes 'no difference' in the first stage.
• Qwen2.5-VL-72B's CSS captioning scores are comparable to proprietary models, yet its LLM-as-a-judge accuracy (24.3) is far below GPT-5-thinking (43.0).

Limitations

• Fine-tuning gains on spatial and temporal categories are modest, leaving a large gap to GPT-5-thinking and humans.
• Several proprietary models' results depend on API access and prompt settings, and some categories are small (e.g., Quantity-Aerial with 49 items).

Experimental setup

Datasets: VLM-SubtleBench validation split MVTEC-AD COCO MIMIC-Diff-VQA ChangeIt CREMA-D RAVDESS AFEW-VA DAiSEE YT8M VLM4D LEVIR-MCI

Benchmarks: VLM-SubtleBench MLLM-CompBench MMAD QAG-360K

Metrics: accuracy cosine similarity score LLM-as-a-judge accuracy

Baselines: MLLM-CompBench Random Guess Human Eval

Builds on: MLLM-CompBench Spot-the-Diff MVTEC-AD MIMIC-Diff-VQA

Closest papers by meaning

• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.90)
• MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs (NeurIPS 2024 · similarity 0.90)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.89)
• VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
• Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.89)