Research map / Vision-language and multimodal / Multimodal LLMs
Visual Agentic AI for Spatial Reasoning with a Dynamic API
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VADAR, a training-free agentic framework where LLM agents dynamically generate a Pythonic API of reusable 3D spatial reasoning functions, and a new Omni3D-Bench benchmark for multi-step 3D VQA.
Problem
VLMs perform poorly on 3D spatial reasoning requiring multi-step grounding and inference, and existing visual program synthesis methods rely on static, human-defined APIs that limit the range of queries they can handle.
Approach
LLM Signature and Implementation agents collaboratively generate new Python API functions as needed for subproblems (depth-first implementation with test-agent feedback), and a Program Agent composes these into programs executed with vision specialists, replacing a fixed DSL with a dynamic one.
Key results
- VADAR reaches 83.0% total accuracy on CLEVR with oracle vision specialists versus 42.6% for ViperGPT and 39.9% for VisProg.
- On Omni3D-Bench VADAR gets 40.4% total accuracy, behind only GPT-4o (42.9%) among evaluated models and ahead of Claude3.5-Sonnet (32.2%) and Gemini1.5-Pro (32.0%).
- VADAR outperforms the no-API single-agent variant by about 6 points on CLEVR-100 (66.7 vs 60.7 with prompting ablations).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VADAR's dynamically generated API instead of VisProg's static API | Program synthesis for CLEVR · CLEVR | accuracy (%): 31.2 -> 53.6 | Helped |
| Use VADAR's dynamically generated API instead of ViperGPT's static API | Program synthesis for Omni3D-Bench · Omni3D-Bench | accuracy (%): 26.7 -> 40.4 | Helped |
| Replace VADAR's vision specialists with oracle variants | VADAR · CLEVR | accuracy (%): 53.6 -> 83.0 | Helped |
| Generate programs directly without a reusable API (ablation) | VADAR on CLEVR₁₀₀ · CLEVR₁₀₀ | accuracy: 64.0 -> 60.7 | Hurt |
| Add a generated API of reusable methods (ablation) | No-API Agent on CLEVR₁₀₀ · CLEVR₁₀₀ | accuracy: 60.7 -> 64.0 | Helped |
| Add weak ICL examples for the Implementation Agent (ablation) | API Agent on CLEVR₁₀₀ · CLEVR₁₀₀ | accuracy: 64.0 -> 65.7 | Helped |
| Add Pseudo ICL instructions and examples to the Implementation and Program Agents (ablation) | API Agent + Weak ICL on CLEVR₁₀₀ · CLEVR₁₀₀ | accuracy: 65.7 -> 66.7 | Helped |
| Use VADAR instead of GPT-4o | Visual reasoning on CLEVR · CLEVR | accuracy (%): 58.4 -> 53.6 | Hurt |
| Use VADAR instead of GPT-4o | Visual reasoning on Omni3D-Bench · Omni3D-Bench | accuracy (%): 42.9 -> 40.4 | Hurt |
| Use VADAR instead of Gemini1.5-Pro | Image-based spatial reasoning · VSI-Bench-img | accuracy: 49.5 -> 50.1 | Helped |
Surprising findings
- On GQA, the monolithic GPT-4o outperforms VADAR (54.9 vs 46.1), so the program-synthesis advantage is specific to 3D reasoning.
- Program synthesis with oracle vision reaches 94.4% on Omni3D-Bench subset, suggesting vision specialists rather than program logic are the bottleneck.
Limitations
- Struggles with queries requiring five or more inference steps.
- Execution accuracy (53.6% on CLEVR) lags program accuracy (83.0%) because of vision specialist errors.
- Program is generated from the query alone without the image, so image content cannot inform program synthesis.
Experimental setup
Datasets: CLEVR Omni3D GQA VSI-Bench
Benchmarks: CLEVR Omni3D-Bench GQA VSI-Bench-img
Metrics: accuracy MRA
Baselines: GPT-4o Claude3.5-Sonnet Llama3.2 Gemini1.5-Pro Gemini1.5-Flash Molmo SpaceMantis ViperGPT VisProg LEFT
Builds on: ViperGPT VisProg LEFT library learning
Closest papers by meaning
- SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (NeurIPS 2026 · similarity 0.91)
- Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.90)
- Geometrically-Constrained Agent for Spatial Reasoning (CVPR 2026 · similarity 0.90)
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.90)
- Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models (ICML 2026 · similarity 0.90)
- ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.