Research map / Vision-language and multimodal / Multimodal LLMs
SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SpatiaLab, a 1,400-item real-world spatial reasoning benchmark with MCQ and open-ended formats across 6 categories and 30 subcategories, showing large gaps between VLMs and humans.
Problem
Existing spatial reasoning benchmarks for VLMs are synthetic, simplified, or narrow in task design and mostly use closed-ended formats, so they fail to capture real-world visual complexity and multi-step spatial reasoning.
Approach
A manually annotated, in-the-wild benchmark spanning six spatial categories and 30 subcategories, evaluated in both multiple-choice and open-ended (LLM-judged) formats against human baselines, with exploratory SFT, CoT, self-reflection, and a multi-agent pipeline (SpatioXolver).
Key results
- On SpatiaLab-MCQ, InternVL3.5-72B reaches 54.93% accuracy versus 87.57% for humans (random 25%).
- On SpatiaLab-Open, GPT-5-mini scores highest at 40.93% versus 64.93% for humans.
- The LLM judge (Gemini-2.5-Flash) reaches Cohen's kappa 0.738 against the majority vote of human annotators.
Surprising findings
- Spatially specialized models (SpaceOm, SpaceThinker, SpaceQwen) do not outperform general models; they reach only about 40-41% on MCQ and 10-13% on open-ended.
- SFT on 40% of the benchmark improves MCQ accuracy but offers little or negative transfer to open-ended tasks; CoT often reduces accuracy.
Limitations
- Relatively small benchmark (1,400 QA pairs) with some subcategories containing only about 25 questions.
- LLM-as-judge for open-ended evaluation, with imperfect agreement with humans (kappa 0.738).
- Some copyright and privacy oversights may remain in web-sourced images.
Experimental setup
Datasets: SpatiaLab
Benchmarks: SpatiaLab-MCQ SpatiaLab-Open
Metrics: accuracy Cohen's kappa Fleiss' kappa
Baselines: GPT-4o-mini Gemini-2.5-Flash InternVL3.5-72B o4-mini SpaceOm SpaceThinker-Qwen2.5VL-3B SpaceQwen2.5-VL-3B-Instruct Human baseline
Builds on: Xolver
Closest papers by meaning
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.95)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.94)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.94)
- SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.93)
- SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.93)
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.