Research map / Vision-language and multimodal / Multimodal LLMs

BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BLINK-Twice, a vision-centric reasoning benchmark of 345 base images across seven visual-illusion challenges, with GPT-4o-edited adversarial pairs and annotated reasoning chains, and evaluates 20 MLLMs zero-shot.

Problem

Existing multimodal reasoning benchmarks mostly test language/knowledge reasoning with visual input as replaceable context, and perception benchmarks test only shallow perception, so vision-grounded reasoning is under-evaluated.

Approach

Build perception-challenging images (visual illusions) requiring detailed observation, pair them with GPT-4o-edited natural adversarial images having opposite answers, and score reasoning chains (CoT-score) on key visual clues and true reality in addition to answer accuracy.

Key results

• Gemini-2.5-pro achieves the best results with Q-Acc 0.667, I-Acc 0.470, and G-Acc 0.269 on BLINK-Twice.
• QVQ-72B reaches a CoT score of 0.438 among open-source models and outperforms Qwen2-VL-72B by about 15% in Q-Acc, but produces over 950 tokens on average versus ~100 for annotated steps.
• Multi-turn re-observation of the image improves weaker models such as Gemini-2.0-flash-thinking and Qwen2-VL-72B, while GPT-4o and Qwen2.5-VL-72B gain little.

Measured effects reported in the paper

Surprising findings

• Reasoning-enhanced QVQ-72B produces redundant over-long reasoning (over 950 tokens on average) despite its accuracy gains.
• No evaluated model answers all four questions in an adversarial group correctly; even the best G-Acc is only 0.269.

Limitations

• Adversarial image generation was limited by OpenAI image policies, yielding only 243 initial samples and 103 retained adversarial samples.
• Reasoning chains and CoT scoring rely on GPT-4o annotation and GPT-4o scoring, with human review but potential hallucination bias.
• Benchmark size is modest (345 base images, 896 VQA questions).

Experimental setup

Datasets: BLINK-Twice

Benchmarks: BLINK-Twice

Metrics: No-Acc Yes-Acc Q-Acc I-Acc G-Acc CoT Score

Baselines: GPT-4o Gemini-2.5-pro o1 QVQ-72B Claude-3.7-sonnet-thinking Qwen2.5-VL-72B InternVL2-40B MM-Eureka-8B

Builds on: NaturalBench Chain-of-thought prompting BLINK

Closest papers by meaning

• Act2See: Emergent Active Visual Perception for Video Reasoning (CVPR 2026 · similarity 0.91)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.91)
• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.91)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.90)