Research map / Vision-language and multimodal / Multimodal LLMs

MentisOculi: Revealing the Limits of Reasoning with Mental Imagery

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MentisOculi, a procedural, stratified benchmark of five multi-step visual reasoning tasks, and finds that explicit visual thoughts (latent, interleaved image generation, video) generally fail to improve frontier model reasoning.

Problem

Whether models can use self-generated mental imagery (latent, interleaved images, or video) to aid multi-step visual reasoning remains unclear, and no rigorous benchmark isolates this capability.

Approach

A procedural benchmark of five visual reasoning tasks (Form Board, Hinge Folding, Paper Fold, Rush Hour, Sliding Puzzle) with five difficulty levels and ground-truth visual chain-of-thought, used to disentangle generation errors from interpretation errors across model families.

Key results

• Gemini 3 Pro is the best model, but most models fail to reliably exceed chance even at Level 1 on most tasks, and all models are at or below chance by Level 5.
• Gemini 3-I and Gemini 2.5-I (image-generating UMMs) generally fail to beat their text-only MLLM counterparts on Rush Hour.
• Oracle ground-truth visual CoT only partially helps UMMs: it lifts Form Board to peak accuracy but only closes the gap to the underlying MLLM on Hinge Folding and Paper Fold.

Surprising findings

• Providing ground-truth visual chains of thought often fails to improve UMM reasoning, indicating interpretation errors in addition to generation errors.
• Generating a Veo 3.1 video reasoning trace costs about $3.2 per sample, over 21x Gemini 2.5-I, while performance is similar.

Limitations

• Benchmark is initial version with 30 samples per level, so small sample sizes limit statistical power.
• Human reference data comes from only five PhD students and only on Rush Hour.
• Latent reasoning baselines fine-tuned on only 1,000 samples on a single model (Qwen2.5-VL-7B), limiting generality.

Experimental setup

Datasets: MentisOculi procedural generators ImageNet-1k

Benchmarks: MentisOculi

Metrics: accuracy chance-normalized performance omission rate response time

Baselines: Gemini 2.5 Flash Gemini 3 Pro GPT-5.1 Qwen3-VL Mirage LatentSketchpad Emu 3.5 Gemini 2.5 Flash Image Gemini 3 Pro Image Veo 3.1 Wan 2.6

Builds on: Mirage LatentSketchpad OpenEvolve

Closest papers by meaning

• Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens (CVPR 2026 · similarity 0.92)
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.91)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.90)
• Imagine While Reasoning in Space: Multimodal Visualization-of-Thought (ICML 2025 · similarity 0.90)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)