Research map / Vision-language and multimodal / Multimodal LLMs

HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a four-level hierarchy of 3D spatial intelligence, builds a ~5M-image spatial VQA dataset, and trains an RGB-D PaliGemma2-3B VLM with metric point maps that reports SOTA on spatial benchmarks.

Problem

VLMs lack a systematic task design for 3D spatial understanding across cognitive levels and large-scale, 3D-grounded training data for it; existing datasets are limited to indoor scenes or lack explicit 3D supervision.

Approach

Organize spatial VQA into four progressive levels (geometric perception, object-level, inter-object relations, abstract reasoning), auto-generate ~2B QA pairs from web and 3D-annotated images, and feed a metric-scale XYZ point map (from MoGe-2 or GT depth) as an extra input branch to the VLM alongside RGB.

Key results

• HiSpatial-3B (RGB-XYZ) reaches 79.28 avg on SpatialRGPT-Quantitative and 85.16 avg on QSpatial-Bench, above GPT-5 (40.47 and 68.45) and Gemini-2.5-Pro (26.57 and 49.92).
• On qualitative benchmarks, HiSpatial-3B scores 97.58 on CV-Bench-3D and 80.71 on EmbSpatial, exceeding the best listed specialist and proprietary baselines.
• On the custom benchmark, HiSpatial-3B reaches 92.18% object-to-camera distance accuracy vs. 47.19% for GPT-5.

Surprising findings

• Removing level 0 and 1 training data hurts level 2 and level 3 performance, indicating lower-level tasks help higher-level reasoning even without explicit CoT.
• Removing levels 1 and 2 causes a larger level-3 drop (-14.51%) than removing levels 0 and 1 (-8.14%).

Limitations

• Generalization limited by template-based instructions and reliance on fixed instruction patterns; weaker on diverse informal language.
• Focuses on basic spatial tasks; Level 3 coverage is incomplete and no RFT is used yet.
• Monocular input only; no multi-view or video spatial reasoning.

Experimental setup

Datasets: KOSMOS-2 (COYO-700M subset) Objects365 CA-1M LLaVA-Next general VQA Omni3D

Benchmarks: SpatialRGPT QSpatial CV-Bench EmbSpatial RoboSpatial 3DSRBench MMBench POPE SEED RealWorldQA

Metrics: accuracy success rate within relative error threshold

Baselines: GPT-5 Gemini-2.5-Pro GPT-4o Claude-3.7-Sonnet Qwen3-VL-8B InternVL3.5-8B RoboRefer-8B-SFT SpatialRGPT-8B MM-Spatial-3B SpatialBot-3B

Builds on: PaliGemma-2 SpatialVLM SpatialRGPT MoGe-2

Closest papers by meaning

• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.92)
• SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.92)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.92)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.92)
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.92)