Research map / Vision-language and multimodal / Multimodal LLMs

CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CoSpace, a benchmark of 2,918 images and 1,626 QA pairs testing continuous space perception (multi-view spatial reconstruction from static-viewpoint panoramas) in 19 VLMs, finding proprietary models are more response-consistent but not much more accurate.

Problem

Existing multi-image and spatial benchmarks do not evaluate whether VLMs can reconstruct a continuous 360-degree space from overlapping views captured from a static viewpoint, which matters for planning and navigation.

Approach

Defines continuous space perception as understanding a scene from sequential, overlapping images taken by rotating in place, and builds a seven-task benchmark (direction, rotation, counting, planning) with order-perturbed pairwise accuracy to measure consistency.

Key results

• Claude-3.7-sonnet achieves the highest average accuracy of 51.97% on CoSpace, while InternVL2_5-78B reaches 45.50% and LLaVA-OneVision-72B only 29.17%.
• Open-source models score near 50% on ROT-Ang, largely by outputting a fixed answer, while humans reach 96.17% on that task.
• Human average ACC_q is 86.25% versus 59.56% for Claude-3.7-sonnet; proprietary models show higher pairwise-accuracy variance than open-source models.

Measured effects reported in the paper

Surprising findings

• Proprietary models show larger accuracy variance across image-order perturbations than open-source models, so the main gap is consistency rather than accuracy.
• Some sub-13B open-source models score higher under shuffled image order than under the correct order on certain tasks.

Limitations

• Only multiple-choice or semi-open-ended formats; fully open-ended evaluation was not conducted.
• Questions generated by GPT-4o and answers annotated by humans, with mostly outdoor Baidu panoramas and a limited indoor HM3D subset.
• Small benchmark size (1,626 QA pairs) with few samples per task, e.g. 300 ROT-Ang questions.

Experimental setup

Datasets: Baidu Map Panorama API images HM3D

Benchmarks: CoSpace

Metrics: ACC_q ACC_p ACC conditional accuracy

Baselines: GPT-4o Gemini-1.5-pro Claude-3.7-sonnet InternVL2_5-78B Qwen2-VL-72B LLaVA-OneVision-72B MiniCPM-V 2.6 Qwen2-VL-7B Mantis-8B InternVL2-8B VILA1.5-8B Idefics3-8B

Builds on: HM3D Habitat-Sim GPT-4o question generation

Closest papers by meaning

• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.90)
• SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness (NeurIPS 2025 · similarity 0.90)
• See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.90)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.89)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.89)