Research map / Vision-language and multimodal / Multimodal LLMs

Thinking with Programming Vision: Towards a Unified View for Thinking with Images

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CodeVision, a code-as-tool framework where an MLLM writes Python to invoke image operations, trained with SFT then RL using a dense process reward, and evaluated on orientation-robustness benchmarks and a new multi-tool benchmark (MVToolBench).

Problem

Current thinking-with-images MLLMs rely on a narrow set of pre-registered tools (mostly crop), are brittle to image orientation changes and natural corruptions, and do not compose multiple different tools across turns.

Approach

Treat code as a universal tool interface so the model can invoke any image operation via generated code, trained with SFT on multi-turn tool trajectories (including error recovery) and RL with a dense multi-component reward (must-use tool bonuses, rollout-based suggested-tool bonus, and constraint penalties against reward hacking).

Key results

• CodeVision-7B reaches 73.4 average on transformed OCRBench, a +17.4 gain over Qwen2.5-VL-7B (56.0).
• On MVToolBench, CodeVision-7B scores 60.1 versus 32.6 for Gemini2.5-Pro, the next best reported model.
• Removing the strategy reward drops CodeVision-7B's MVToolBench score from 60.1 to 50.7.

Surprising findings

• Simple rotation or flip can reduce state-of-the-art MLLM performance by up to 80% in the diagnostic study.
• RL trained from the base model without SFT cold start fails to achieve meaningful improvement.

Limitations

• Focuses on a limited toolset, primarily orientation correction and cropping.
• Fine-grained crop coordinate prediction remains imprecise, as shown in failure cases.
• Learning has not saturated; scaling of data and tasks is left for future work.

Experimental setup

Datasets: HierText OCRBench ChartQAPro V* HRBench

Benchmarks: OCRBench (rotated/flipped) ChartQAPro (rotated/flipped) V* HRBench4k HRBench8k MVToolBench

Metrics: accuracy

Baselines: GPT-4o Gemini2.5-Pro Qwen2.5-VL-32B Qwen2.5-VL-72B Qwen3-VL-30B-Thinking Qwen3-VL-235B-Thinking InternVL3.5-30B InternVL3.5-241B Thyme Qwen2.5-VL-7B Qwen3-VL-8B-Thinking Qwen3-VL-32B-Thinking

Builds on: GRPO Thyme OpenAI o3 thinking with images

Closest papers by meaning

• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.92)
• What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning (AAAI 2026 · similarity 0.91)
• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization (CVPR 2026 · similarity 0.90)
• When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.90)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.90)