Research map / Large language models / Agents, code and math
Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visualization-of-Thought (VoT) zero-shot prompting, which has LLMs draw text-form 2D grid states after each reasoning step, and evaluates it on natural-language, visual navigation and visual tiling tasks with GPT-4, GPT-4V, GPT-3.5 and LLaMA3.
Problem
LLMs' spatial reasoning is underexplored; language-based reasoning may not capture spatial state, and existing MLLMs underperform on multi-hop spatial tasks in grid worlds.
Approach
Prompt the LLM to visualize its state after each reasoning step as an ASCII/emoji grid (text-form mental image), so subsequent steps are conditioned on the visualized state (visual state tracking), without few-shot demos or external image generators.
Key results
- GPT-4 VoT improves natural-language navigation over GPT-4 w/o Viz by 23.5% (59.00 vs 35.50) and visual tiling to 63.94 vs 54.15 for GPT-4 CoT.
- On visual navigation route planning, GPT-4 VoT reaches 14.72% success rate vs 5.65% for GPT-4V CoT, and next-step prediction 55.28 vs 48.61 for GPT-4 CoT.
- LLAMA3-70B VoT improves over CoT on route planning success (5.85 vs 2.62) and next-step prediction (54.09 vs 49.01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VoT instead of CoT | GPT-4 · Visual Navigation route planning | completing rate: 37.02 -> 40.77 | Helped |
| Use VoT instead of CoT | GPT-4 · Visual Navigation route planning | success rate: 9.48 -> 14.72 | Helped |
| Use VoT instead of CoT | GPT-4 · Visual Navigation next-step prediction | accuracy: 48.61 -> 55.28 | Helped |
| Use VoT instead of CoT | GPT-4 · Visual Tiling | accuracy: 54.15 -> 63.94 | Helped |
| Use VoT instead of CoT | GPT-4 · Natural-Language Navigation | accuracy: 54.00 -> 59.00 | Helped |
| Use VoT instead of CoT | GPT-3.5 turbo · Visual Navigation route planning | success rate: 2.62 -> 1.61 | Hurt |
| Use VoT instead of CoT | GPT-3.5 turbo · Visual Navigation next-step prediction | accuracy: 17.42 -> 13.10 | Hurt |
| Use VoT instead of CoT | GPT-3.5 turbo · Visual Tiling | accuracy: 44.10 -> 47.99 | Helped |
| Use VoT instead of CoT | LLAMA3-70B-Instruct · Visual Navigation route planning | completing rate: 19.90 -> 30.24 | Helped |
| Use VoT instead of CoT | LLAMA3-70B-Instruct · Visual Tiling | accuracy: 56.41 -> 56.03 | No clear effect |
| Use VoT instead of CoT | GPT-4 · Natural-Language Navigation, ring size 12 | accuracy: 52.5% VS 49.5% | Hurt |
| Add the phrase “use ascii-art to visualize” to the prompt (ablation) | GPT-4 · Natural-Language Navigation | accuracy: +3.5% | Helped |
Surprising findings
- GPT-4 CoT already shows visual state tracking behavior on several tasks without explicit visualization prompting.
- VoT underperforms CoT on a ring-based navigation task where logical arithmetic reasoning suffices (49.5% vs 52.5%).
Limitations
- Visual state tracking and accuracy of generated visualizations are low (about 24-26% state accuracy), and performance depends heavily on prompt wording.
- Less capable models (GPT-3.5, LLaMA3-8B) show random guessing and unstable gains; results limited to 2D grid mental images.
- Evaluated on small synthetic tasks with limited sample sizes.
Experimental setup
Datasets: Natural-language navigation maps (200, 3x3) Visual navigation synthetic maps (496 maps, 2520 QA) Visual tiling polyomino puzzles (796 QA)
Benchmarks: Natural-language navigation Visual navigation (route planning, next step prediction) Visual tiling
Metrics: accuracy average completing rate success rate complete tracking rate partial tracking rate compliance spatial visualization accuracy
Baselines: GPT-4 CoT GPT-4 w/o Viz GPT-4V CoT GPT-3.5 CoT LLAMA3-8B CoT LLAMA3-70B CoT
Builds on: chain-of-thought prompting visuospatial sketchpad
Closest papers by meaning
- Imagine While Reasoning in Space: Multimodal Visualization-of-Thought (ICML 2025 · similarity 0.92)
- Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing (NeurIPS 2025 · similarity 0.91)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.90)
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.90)
- EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs (CVPR 2026 · similarity 0.89)
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.