Research map / Vision-language and multimodal / Multimodal LLMs

Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Perception Programs (P²), a training-free method that rewrites vision-tool outputs (depth, flow, correspondence, detection) into compact YAML-like language-native summaries, improving MLLM performance on six BLINK perception sub-tasks.

Problem

MLLMs fail to benefit from dense, pixel-level vision tool outputs (depth, flow, correspondence) because these raw representations misalign with language-native reasoning.

Approach

Rewrite each tool output into a structured, grid-level symbolic summary (location, read-out, optional label and relations) serialized as YAML-like text, so off-the-shelf MLLMs can read the visual cue without training or extra tool calls.

Key results

• GPT-5 Mini with P² rises from 41.35% to 86.47% on multi-view reasoning and from 52.42% to 81.45% on HardBLINK depth.
• Qwen3VL-4B with P² reaches 93.98% on multi-view reasoning versus 67.66% with raw tool output.
• P² turns Gemini 2.5 Pro's raw-tool regression of 14.29 points on multi-view reasoning into a 63.91% score (vs 48.12% standard).

Surprising findings

• Appending raw tool outputs can be neutral or harmful; Gemini 2.5 Pro drops on multi-view reasoning with raw tool use.
• GPT-5 Mini cannot reconstruct depth orderings from depth maps as grid resolution increases (Kendall's tau approaches zero), and for correspondence often copies input coordinates.

Limitations

• Evaluated only on six BLINK sub-tasks with clear tool surrogates; broader 3D or general VQA not studied.
• Errors from upstream tools propagate into the program; no calibration of conflicting tools.
• Tool sequencing and dynamic tool selection are not explored.

Experimental setup

Datasets: BLINK HardBLINK

Benchmarks: BLINK multi-view reasoning HardBLINK depth BLINK visual correspondence BLINK jigsaw BLINK object localization BLINK semantic correspondence

Metrics: accuracy

Baselines: Visual Sketchpad MMFactory Thyme LATTE Mirage Aurora ReVPT ViGoRL OVR TULIP PerceptionLM Zebra-CoT

Builds on: BLINK Visual Sketchpad Thyme LATTE

Closest papers by meaning

• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.93)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.93)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.92)
• Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.92)
• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.91)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.91)