Research map / Vision-language and multimodal / Multimodal LLMs

SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpatialClaw, a training-free spatial reasoning agent that uses code (a persistent Python kernel with perception tools like Depth Anything 3 and SAM3) as the action interface, evaluated on 20 spatial benchmarks across six open VLM backbones.

Problem

Tool-augmented VLM agents for spatial reasoning are bounded by their action interface; single-pass code commits to a full plan before observing results, and structured tool-calls offer limited compositional flexibility for open-ended 3D/4D spatial reasoning.

Approach

Treat code as the action interface: a persistent, stateful Python kernel pre-loaded with frames and perception/geometry primitives, where the VLM agent writes one cell per step conditioned on all prior outputs (text, variables, rendered images), enabling iterative composition and revision without any training.

Key results

• SpatialClaw with Qwen3.5-397B reaches 59.9% average over 20 benchmarks in the Gemma4-31B comparison, outperforming SpaceTools by +11.2 points.
• Across six backbones, SpatialClaw improves average accuracy over the no-tool baseline (e.g., Qwen3.6-27B 55.0 to 62.7; Gemma4-26B-A4B 48.0 to 54.3).
• In the interface comparison with Gemma4-31B, SpatialClaw averages 59.9 vs 56.7 for structured tool-call and 55.2 for single-pass code.

Surprising findings

• Removing all predefined utility functions causes only a small drop (59.9 to 56.4 average), suggesting on-the-fly numpy/scipy computation compensates.
• Without perception tools (SAM3/DA3), the code-only interface still gains +2.7 points over the no-tool baseline.

Limitations

• Relies on closed-source Gemini-3.1-Pro as judge for failure analysis and attribution.
• Evaluation subsampled to 1,000 samples per benchmark and 500 for ablations; baselines are not all tuned for these benchmarks.
• Geometric reasoning and perception errors remain the main failure modes.

Experimental setup

Datasets: ERQA Omni3D OmniSpatial SPBench MindCube MMSI SPAR-Bench MMSI-Video OSI-Bench PAI-Bench VSI-Bench-U VSTI-Bench

Benchmarks: ERQA Omni3D OmniSpatial SPBench MindCube MMSI SPAR-Bench BLINK SpatialTree ViewSpatial MMSI-Video OSI-Bench

Metrics: accuracy mean relative accuracy (MRA) VCI

Baselines: No-tool baseline Single-pass code Structured tool-call VADAR pySpatial SpaceTools

Builds on: CodeAct VADAR pySpatial SpaceTools

Closest papers by meaning

• Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning (NeurIPS 2026 · similarity 0.92)
• Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models (ICML 2026 · similarity 0.92)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
• SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.91)
• Visual Agentic AI for Spatial Reasoning with a Dynamic API (CVPR 2025 · similarity 0.91)
• SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.91)