Research map / Vision-language and multimodal / Multimodal LLMs
Geometrically-Constrained Agent for Spatial Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GCA, a training-free two-stage agent that has a VLM formalize each spatial query into a reference-frame and objective constraint, then solve it via ReAct-style tool calls (VGGT, detection, pose, code), reaching 47.6 on MMSI-Bench and 64.8 average across five spatial benchmarks.
Problem
VLMs lose fine-grained geometry when reasoning in semantic text space, and existing training-based and tool-integrated spatial methods either learn from imperfect oracles or leave the VLM's planning unconstrained, producing geometrically flawed plans.
Approach
Decouple the VLM into a semantic analyst that writes a formal task constraint (reference frame plus objective) and a task solver that performs tool-based geometric computation strictly within that constraint, with knowledge-augmented code generation from a fixed formula library.
Key results
- GCA with Qwen3-VL-Thinking reaches 47.6% on MMSI-Bench, versus 36.9% for the best foundation baseline Gemini-2.5-Pro and 25.4% for SpatialLadder.
- Average accuracy across the benchmarks is reported as 64.8%, with 64.2% on MindCube-tiny and 65.1% on OmniSpatial.
- Removing the reference frame constraint drops MMSI-Bench accuracy by 6.6 points, while removing the objective constraint drops it by 1.2 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add formal task constraint C_task (reference frame + objective) before tool-based computation (ablation) | Qwen3-VL-Thinking agent · MMSI-Bench | accuracy (All): +7.5 points (40.1 to 47.6) | Helped |
| tool integration added over CoT-only (ablation) | Qwen3-VL-Thinking · MMSI-Bench | accuracy (All): +4.2 points (32.6 to 36.8) | Helped |
| knowledge-augmented code generation (KACG) (ablation) | tool-integrated agent · MMSI-Bench | accuracy (All): +1.9 points (36.8 to 38.7) | Helped |
| VLM-managed tool feedback and ambiguity resolution (ablation) | tool-integrated agent · MMSI-Bench | accuracy (All): +1.4 points (38.7 to 40.1) | Helped |
| tell VLM the task constraint as a text prompt without tools (CoT only) (ablation) | Qwen3-VL-Thinking CoT · MMSI-Bench | accuracy (All): +0.9 points (32.6 to 33.5) | No clear effect |
| prompt VLM to pay attention to reference frame and objective (unconstrained tools) (ablation) | unconstrained tool integration · MMSI-Bench | accuracy (All): negligible improvement | No clear effect |
| remove objective constraint C_O (keep C_R) (ablation) | GCA · MMSI-Bench | accuracy (All): -1.2 points (47.6 to 46.4 when removed) | Helped |
| remove reference frame constraint C_R (keep C_O) (ablation) | GCA · MMSI-Bench | accuracy (All): -6.6 points (47.6 to 41.0 when removed) | Helped |
| GCA paradigm applied to foundation VLMs | Gemini-2.5-Pro · MMSI-Bench | accuracy (All): 36.9 to 55.0 (+49% relative) | Helped |
| GCA paradigm applied to foundation VLMs | GPT-4o · MMSI-Bench | accuracy (All): +19% relative | Helped |
| GCA agent vs training-based and tool-integrated baselines on out-of-domain benchmarks | GCA with Qwen3-VL-Thinking · SPBench and multi-view benchmarks | accuracy | Mixed |
Surprising findings
- Adding the task constraint as a textual hint to a CoT-only VLM yields only a 0.9-point gain, indicating the constraint must structure tool-based computation rather than act as a prompt.
- Training-based spatial VLMs such as SpatialLadder perform well in-domain on SPBench but fall well behind on out-of-domain benchmarks.
Limitations
- Iterative tool calls and VLM interactions are computationally more costly than end-to-end CoT reasoning.
- Toolbox is designed for image-based inputs, so video and temporal reasoning are not covered.
- Reference frames for abstract regions (e.g., living room) rely on proxies that can accumulate errors.
Experimental setup
Datasets: MMSI-Bench MindCube-tiny OmniSpatial SPBench CV-Bench
Benchmarks: MMSI-Bench MindCube-tiny OmniSpatial SPBench CV-Bench
Metrics: accuracy
Baselines: Qwen3-VL-Thinking GLM-4.5V GPT-4o Gemini-2.5-Pro SpatialLLM Spatial-MLLM SpatialLadder SpaceR Video-R1 RoboBrain-2.0 VILASR VLaser
Builds on: ReAct LLM+P ReKep LogicLM VGGT
Closest papers by meaning
- Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.92)
- SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (NeurIPS 2026 · similarity 0.91)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces (ICML 2026 · similarity 0.90)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Visual Agentic AI for Spatial Reasoning with a Dynamic API (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.