Research map / Vision-language and multimodal / Multimodal LLMs

Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains PERIA, a tool-augmented Qwen3-VL-Thinking agent with perception and interaction tools for spatial reasoning, using SFT on synthesized tool trajectories followed by RL with a new observation-relaxed GIGPO variant (OR-GIGPO).

Problem

Vision-language models struggle with spatial reasoning tasks that require active evidence acquisition and multi-step visual interaction, and open-source VLMs do not reliably use available visual tools without tool-use training.

Approach

Combine a unified perception-interaction tool sandbox with SFT trajectories (explore-and-exploit synthesis plus reasoning diversification) and an RL method, OR-GIGPO, that groups step-level anchors by relaxed (hash or SequenceMatcher-similar) tool observations instead of exact matches.

Key results

• PERIA-8B averages 54.1% on in-distribution benchmarks versus 44.1% for Qwen3-8B (+10.0 points) and 44.4% overall, comparable to GPT-5 (46.8% overall).
• OR-GIGPO outperforms GRPO by 2.4 points and DAPO by 11.1 points on in-distribution average.
• Without tools at inference, PERIA-8B drops to 44.4% in-distribution average.

Surprising findings

• Open-source Qwen3-VL-Thinking models often degrade when given raw tool access without tool-use training.
• DAPO performs worse than the untrained baseline on some in-distribution tasks in this setting.

Limitations

• Main experiments focus on the 8B model due to computational constraints; larger backbones not validated.
• Inference limited to at most 10 interaction turns, which may not suffice for longer-horizon tasks.
• Relies on proprietary GPT-5 for trajectory synthesis and GPT-5-Mini as judge.

Experimental setup

Datasets: MapTrace ReasonMap ReasonMap-Plus Visual Probing MapQA

Benchmarks: MapTrace ReasonMap ReasonMap-Plus Visual Probing Easy Visual Probing Medium Visual Probing Hard MapEval BabyVision Ball Tracking Paper Folding Cube 3-View Reasoning Real-world Spatial Reasoning

Metrics: accuracy

Baselines: GPT-5 Gemini-2.5-Pro Gemini-2.5-Flash Qwen3-VL-235B-A22B-Thinking InternVL3.5-8B VTool-R1-7B R1-OneVision-7B Mini-o3

Builds on: GIGPO GRPO DAPO ReAct Verl-Tool

Closest papers by meaning

• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (NeurIPS 2026 · similarity 0.92)
• Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models (ICML 2026 · similarity 0.90)
• SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.90)
• Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.90)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.89)