Research map / Vision-language and multimodal / Multimodal LLMs

RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 30 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RoboSpatial, a ~3M-QA spatial reasoning dataset built from 3D-annotated indoor and tabletop scans with reference-frame-aware questions, and shows that fine-tuning 2D and 3D VLMs on it improves spatial reasoning and real-robot manipulation.

Problem

Vision-language models lack spatial understanding needed for robotics, since their training data rarely covers reference frames (ego, world, object) or robotics-relevant relations such as placement context and compatibility.

Approach

Automatically generate spatial QA pairs from existing 3D-annotated scene and tabletop datasets using oriented 3D bounding boxes and camera poses, covering three relation types (context, compatibility, configuration) across three reference frames, then fine-tune both 2D and 3D VLMs on them.

Key results

• LEO fine-tuned on RoboSpatial reaches 71.9% total accuracy on RoboSpatial-Val vs 42.8% for the base LEO model.
• VILA-1.5-8B rises from 40.2% to 63.9% on RoboSpatial-Val after fine-tuning; RoboPoint rises from 38.9% to 70.6%.
• In real-robot experiments, LLaVA-NeXT success rises from 23.7% to 52.6% with RoboSpatial training, above GPT-4o (46.9%).

Measured effects reported in the paper

Surprising findings

• Zero-shot GPT-4o performs comparably to RoboSpatial-trained RoboPoint in robot experiments, which the authors attribute to stronger language understanding.
• Fine-tuning on RoboSpatial slightly lowered RoboPoint's performance in some robot tasks, possibly due to mixing with RoboPoint's own training data.

Limitations

• The top-down occupancy map cannot handle containment relations such as fitting inside or under objects.
• Robot experiments use a modular pipeline with a separate motion planner rather than end-to-end policy learning.
• Data covers only indoor and tabletop household scenes without humans or animals; tabletop diversity is limited.

Experimental setup

Datasets: ScanNet Matterport3D 3RScan HOPE GraspNet-1B EmbodiedScan

Benchmarks: RoboSpatial-Val RoboSpatial-Home BLINK-Spatial SpatialBench MMMU MME MMBench

Metrics: accuracy success rate

Baselines: VILA LLaVA-NeXT SpaceLLaVA RoboPoint 3D-LLM LEO Molmo GPT-4o

Builds on: SpatialVLM SpatialRGPT RoboPoint LLaVA-NeXT LEO

Closest papers by meaning

• RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics (NeurIPS 2025 · similarity 0.92)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.91)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
• Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.91)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)