Research map / Vision-language and multimodal / Multimodal LLMs

Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Astra couples an RL-trained Qwen3-VL-8B agent (Astra-VL) that issues camera-motion queries with a Bagel-based action-conditioned world simulator (Astra-WM) fine-tuned via view consistency tuning, improving MMSI-Bench and MindCube accuracy.

Problem

Vision-language models struggle to infer unobserved layouts, keep cross-view consistency, and reason from alternative viewpoints when only limited egocentric observations are available for spatial reasoning.

Approach

Treat spatial reasoning as interactive evidence acquisition: a VLM policy decides when and where to request imagined novel views via natural-language camera-motion actions from a world simulator, trained with a two-phase simulator-in-the-loop RL curriculum that rewards selective imagination relative to a no-tool baseline.

Key results

• Astra full framework improves Qwen3-VL-8B from 29.8 to 38.8 on MMSI-Bench and from 36.8 to 42.7 on MindCube.
• Astra-WM (60k) lifts simulator-augmented Gemini-3-Flash on MMSI-Bench from 45.1 to 49.5 versus off-the-shelf Bagel at 45.8.
• Single-stage tool-gain reward collapses tool-call rate to 4.9%, while usage bonus alone yields 98.1% tool rate; two-phase curriculum reaches 61.5% tool rate with 38.8 overall.

Surprising findings

• Naively connecting Qwen3-VL-8B to the world simulator with forced tool use degrades performance below direct answering (29.8 to 28.6 on MMSI-Bench).
• Forced tool use hurts object- and region-centric relations even while improving camera-centric relations.

Limitations

• Open-source policy can collapse to direct answering or overuse the simulator depending on reward design.
• Simulator may produce visually plausible but spatially inconsistent or unhelpful views, and the policy can over-trust them.
• Sparse exact-match reward does not capture partially useful observations.

Experimental setup

Datasets: IsaacSim ScanNet++ ScanNet Matterport3D DL3DV ARKitScenes SenseNova-800K VST-500K Hard-UMMQA MMSI-Bench MindCube

Benchmarks: MMSI-Bench MindCube-tiny

Metrics: exact-match accuracy pose consistency content consistency tool-call rate calls per row

Baselines: Qwen3-VL-8B-Instruct Qwen3-VL-30B-Instruct Bagel-7B-MoT Gemini-3-Flash GLM-4.5V Gemini-2.5-Pro GPT-4o SpatialLadder VILASR Spatial-MLLM

Builds on: GRPO Bagel Qwen3-VL veRL thinking with images

Closest papers by meaning

• MindJourney: Test-Time Scaling with World Models for Spatial Reasoning (NeurIPS 2025 · similarity 0.90)
• 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models (ICML 2026 · similarity 0.90)
• Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens (CVPR 2026 · similarity 0.90)
• Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.90)
• Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models (ICML 2026 · similarity 0.90)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (NeurIPS 2026 · similarity 0.90)