Research map / Vision-language and multimodal / Multimodal LLMs
Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Hyperphantasia, a procedurally generated 1,200-sample synthetic benchmark of four puzzles testing mental visualization in MLLMs, shows large gaps to humans, and uses GRPO on Qwen-VL 2.5 7B to improve extrapolation.
Problem
Current MLLM benchmarks mostly test passive perception, not the active internal construction and manipulation of visual representations (mental visualization) needed for prediction and problem solving.
Approach
A synthetic, procedurally generated benchmark with interpolation (Seven Segments, Connect the Dots) and extrapolation (Linear and Parabolic Trajectory) puzzles at three difficulty levels, plus an RL study showing that training on moderately difficult, diverse extrapolation puzzles transfers to harder and novel puzzles.
Key results
- Claude 3.7 Sonnet reaches 60% on Easy Linear Trajectory but drops to near random on Medium and Hard, while human accuracy averages 84.22% on Hard-level tasks compared with MLLM mean accuracy around 13-58%.
- Gemini 3 pro preview achieves the best mean accuracy (67.25 Easy, 63.00 Medium, 57.75 Hard), and o4-mini is the only other model above 50% on Seven Segments Easy-level tasks at 83%.
- RL with GRPO on Parabolic Trajectory Mix data raises Qwen-VL 2.5 7B to 44% on Hard Parabolic Trajectory from a 18% base, and the Medium Parabolic model reaches 40% on Hard.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on Easy Linear Trajectory puzzles (ablation) | Qwen-VL 2.5 7B · Linear Trajectory Easy vs Medium | accuracy (%): 27 -> 95; 18 -> 9 | Mixed |
| Train on Medium Parabolic Trajectory puzzles (ablation) | Qwen-VL 2.5 7B · Parabolic Trajectory Hard | accuracy (%): 18 -> 40 | Helped |
| Train on Easy and Medium Parabolic Trajectory puzzles (ablation) | Qwen-VL 2.5 7B · Parabolic Trajectory Hard | accuracy (%): 18 -> 44 | Helped |
| Train on Easy and Medium Parabolic Trajectory puzzles (ablation) | Qwen-VL 2.5 7B · Linear Trajectory Easy | accuracy (%): 27 -> 44 | Helped |
| Train on a diverse Hard Mix without Easy Linear examples (ablation) | Qwen-VL 2.5 7B · Extrapolation tasks | accuracy (%): 37; 36; 32; 35; 35; 32 | Mixed |
| Add explicit edges to the Seven Segments image (ablation) | GPT-4o · Seven Segments Easy | accuracy (%): 4 -> 76 | Helped |
| Add explicit edges to the Seven Segments image (ablation) | Gemini 2.5 Pro · Seven Segments Easy | accuracy (%): 51 -> 98 | Helped |
| Vary prompt wording and formatting (ablation) | MLLMs · Linear Trajectory Easy | accuracy (%): Qwen VL 2.5 7B: 27; 32; 26; 23; 25 | No clear effect |
| Add 3-shot examples (ablation) | o4-mini · Linear Trajectory Easy | accuracy (%): 43 -> 40 | Hurt |
| Add 3-shot examples (ablation) | Qwen-VL 2.5 32B · Linear Trajectory Easy | accuracy (%): 40 -> 31 | Hurt |
| Add 3-shot examples (ablation) | Qwen-VL 2.5 7B · Linear Trajectory Easy | accuracy (%): 27 -> 30 | Helped |
| Present Seven Segments as text instead of an image (ablation) | o4-mini · Seven Segments Easy | accuracy (%): 83 -> 88 | Helped |
Surprising findings
- Adding explicit edge cues to Seven Segments images still leaves most models at 0% accuracy, indicating a visual recognition failure beyond mental visualization.
- Several models output a fixed answer regardless of input (e.g., '012' for Qwen-VL 2.5 7B on 32 of 100 Easy puzzles).
Limitations
- Benchmark covers only four puzzle types, a narrow subset of mental visualization
- Seven Segments can be solved from memorized edge patterns, so it was excluded from RL
- Small human study sample and the paper notes trivial-looking tasks where models fail on out-of-distribution visual format
Experimental setup
Datasets: DomainNet Clipart subset
Benchmarks: Hyperphantasia
Metrics: accuracy
Baselines: o4-mini GPT-4o Gemini 2.5 Pro Claude 3.7 Sonnet Qwen-VL 2.5 LLaMA 3.2 LLaVA-OneVision Molmo Deepseek-VL2
Builds on: GRPO Qwen-VL 2.5
Closest papers by meaning
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.90)
- MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.89)
- Imagine While Reasoning in Space: Multimodal Visualization-of-Thought (ICML 2025 · similarity 0.89)
- Do multimodal models imagine electric sheep? (NeurIPS 2026 · similarity 0.88)
- Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models (NeurIPS 2024 · similarity 0.88)
- SpatialViz-Bench: A Cognitively-Grounded Benchmark for Diagnosing Spatial Visualization in MLLMs (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.