Research map / Vision-language and multimodal / Multimodal LLMs

SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpatioLM, a frozen-VLM framework with a plug-and-play Spatio-Vision side module (~0.3B params) that elicits 3D geometry from visual tokens using pseudo depth/camera supervision, improving spatial perception and understanding while preserving general capabilities.

Problem

VLMs are strong at commonsense reasoning but weak at visual spatial and metric 3D reasoning; existing fixes need extra 3D inputs, external spatial encoders, or invasive fine-tuning that degrades general abilities.

Approach

Freeze the VLM and add a ControlNet-style side module (Spatio-Vision Blocks with alternating frame/global attention) fed by intermediate ViT tokens, injected into frozen LM blocks via zero-initialized projections, trained with language loss plus distillation from Depth Anything V3 and dense depth/ray-map supervision.

Key results

• SpatioLM (SenseNovaSI-8B) reaches 71.6 on VSI-Bench, the first reported model above 70 in the paper, and 101.8 CIDEr on ScanQA val.
• On MD-S, SpatioLM (SenseNovaSI-8B) scores 83.5 vs 52.6 for SenseNovaSI-8B; DA-2K accuracy 83.8.
• On LIBERO, SpatioLM-VLA0 improves from 79.2% to 91.0% over SenseNovaSI-VLA0 in the discrete setting; SpatioLM-OFT reaches 96.3% vs 95.3%.

Surprising findings

• Spatial-MLLM drops 23% to 67% on general benchmarks, while SpatioLM stays near baseline and gains about 12% on VideoMMMU.
• SpatioLM slightly degrades on motion-intensive MVBench subtasks such as action counting and moving direction.

Limitations

• Training depends on pseudo-labels from Depth Anything V3, so teacher noise caps 3D comprehension.
• Geometric supervision emphasizes static structure, degrading motion-intensive temporal subtasks.
• Not suitable for safety-critical deployment without further verification.

Experimental setup

Benchmarks: MD-S MD-M DA-2K DR VSI-Bench ScanQA SQA3D VideoMME VideoMMMU MVBench MMMU LIBERO

Metrics: delta accuracy absolute relative error VSI-Bench average accuracy BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr exact match success rate

Baselines: Spatial-MLLM VST-7B-SFT SenseNovaSI-8B InternVL3.5-8B Qwen3-VL-235B-A22B Qwen2.5-VL-72B DepthLM LoRA

Builds on: ControlNet VGGT Depth Anything V3 DepthLM InternVL3.5

Closest papers by meaning

• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
• SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.90)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.90)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)
• HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models (CVPR 2026 · similarity 0.90)