Research map / Vision-language and multimodal / Multimodal LLMs

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Visual Sketchpad, a framework letting multimodal LMs draw intermediate visual sketches via code-executed tools (plotting lines, boxes, marks) as a visual chain of thought for reasoning on vision tasks.

Problem

Multimodal language models reason only through text intermediate steps and existing tool-use paradigms do not let them create visual intermediate artifacts, which limits performance on tasks like geometry, maps, and fine-grained visual perception.

Approach

Equip LMs with a sketchpad: the model writes code to draw auxiliary visual artifacts (lines, marks, boxes, plots) on input images, observes the rendered sketches, and iterates, treating sketches as a visual chain of thought rather than text-only reasoning steps.

Key results

• Visual Sketchpad with GPT-4o improves over base GPT-4o on several vision tasks, with reported gains on BLINK and geometry benchmarks.
• Sketching helps fine-grained perception and math geometry reasoning in a training-free manner.

Limitations

• Performance depends on the underlying LM's ability to write correct drawing code and interpret sketches.
• Evaluated on a limited set of tasks and benchmarks.

Experimental setup

Benchmarks: VSP BLINK MathVista Geometry (geometry diagrams)

Metrics: accuracy

Baselines: GPT-4o GPT-4 Turbo Gemini Claude 3 Opus

Builds on: chain-of-thought visual programming ViperGPT

Closest papers by meaning

• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing (NeurIPS 2025 · similarity 0.91)
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.90)
• Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.89)
• SketchThinker-R1: Towards Efficient Sketch-Style Reasoning in Large Multimodal Models (ICLR 2026 · similarity 0.89)
• Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.88)
• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.88)