Research map / Vision-language and multimodal / Multimodal LLMs

Visual Planning: Let's Think Only with Images

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Visual Planning, where models plan purely by generating image sequences, and VPRL, a two-stage GRPO-based RL method on the autoregressive Large Vision Model (7B), which outperforms text-based planning on FrozenLake, Maze, and MiniBehavior.

Problem

Language-based reasoning in MLLMs is ill-suited for vision-first spatial planning tasks due to a modality gap when grounding visual states into text; the paper asks whether planning can be done purely in visual states.

Approach

Plan by autoregressively generating sequences of images (visual states) with a vision-only model, trained first by random-trajectory initialization (Stage 1, VPFT-style) and then optimized with GRPO using a progress reward that penalizes invalid transitions and rewards progress toward the goal (Stage 2).

Key results

• VPRL achieves 91.6% EM / 93.2% PR on FrozenLake, 74.5% EM on Maze, and 75.8% EM on MiniBehavior, averaging 80.6% EM versus 53.6% for the best text-based SFT Qwen2.5-VL-7B.
• Gemini 2.5 Pro (thinking) reaches 43.7% average EM, while Gemini 2.0 Flash variants stay at or below 12.8%.
• On OOD 7x7 FrozenLake, VPRL reaches 20.4% EM versus 9.6% for VPFT.

Surprising findings

• RL on text-based planning with the VPRL progress reward or PR reward underperforms the SFT text baseline (e.g., 54.4% vs 68.6% EM on FrozenLake for 7B).
• Stage 1 random-trajectory initialization alone yields near-random behavior, yet is needed to preserve exploration for GRPO.

Limitations

• Only the 7B LVM is publicly available, so model scale and comparisons with native multimodal generators are limited.
• Rule-based pixel-level dynamics interpreter is tailored to simple grid environments and may not extend to complex visual structures.
• Image generation adds inference overhead compared to text responses.

Experimental setup

Datasets: FrozenLake (OpenAI Gym, synthetic grids) Maze (self-collected synthetic) MiniBehavior

Benchmarks: FrozenLake Maze MiniBehavior

Metrics: exact match progress rate

Baselines: Gemini 2.0 Flash Gemini 2.5 Pro Qwen 2.5-VL-Instruct-7B SFT Qwen 2.5-VL-Instruct-7B GRPO VPFT

Builds on: GRPO DeepSeek-R1 LoRA VQGAN

Closest papers by meaning

• FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.91)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
• World-aware Planning Narratives Enhance Large Vision-Language Model Planner (NeurIPS 2025 · similarity 0.91)
• AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning (ICLR 2026 · similarity 0.90)
• Evaluating Vision-Language Models as Evaluators in Path Planning (CVPR 2025 · similarity 0.90)
• Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.90)