Research map / Vision-language and multimodal / Multimodal LLMs
SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies 3D-informed data, vision encoder, and multi-stage training for LMMs, producing SpatialLLM (LLaVA-style, CLIP+DINOv2 encoder, Llama3-8B) that reaches 62.7% on the new SpatialVQA benchmark, above GPT-4o at 54.0%.
Problem
Current large multimodal models lack 3D spatial reasoning, especially about object 3D orientation, because 3D-aware training data is scarce and designs are biased toward 2D data.
Approach
A compound design that jointly varies 3D-informed data (probing data with 3D location/orientation from ImageNet3D and OpenImages, plus 3D-informed instruction data), vision encoder mixing, LLM choice, and which training stage receives 3D data, yielding a roadmap for spatially-intelligent LMMs.
Key results
- SpatialLLM reaches 62.7% average accuracy on SpatialVQA, versus 54.0% for GPT-4o and 52.2% for the best spatial-centric model SpatialVLM-13B.
- Adding 3DI-Ft1M instruction tuning gives a 10.7-point gain over the mixed-encoder Llama3 model without it (49.0 to 59.7).
- Using ImageNet3D-derived probing data (3DI-Pb-IN166K) in stage 1 adds about 3 points over OpenImages-derived probing data (60.0 to 62.7 in the roadmap comparison).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a DINOv2 encoder alongside CLIP (ablation) | LLaVA-v1.5 with Vicuna-7B · SpatialVQA | Avg. accuracy: 0.3% | Helped |
| replace Vicuna-7B with Llama3-8B (ablation) | CLIP + DINOv2 model · SpatialVQA | Avg. accuracy: 1% | Helped |
| add 3DI-Ft1M to Stage 2 instruction tuning (ablation) | CLIP + DINOv2, Llama3-8B · SpatialVQA | Avg. accuracy: 10.7% | Helped |
| use 3DI-Pb-IN166K for Stage 1 multimodal alignment (ablation) | CLIP + DINOv2, Llama3-8B, Stage 2 3DI-Ft1M · SpatialVQA | Avg. accuracy: additional 3% | Helped |
| use 3DI-Pb-IN166K instead of 3DI-Pb-OI1M for Stage 1 alignment (ablation) | CLIP + DINOv2, Llama3-8B, Stage 2 3DI-Ft1M · SpatialVQA | Avg. accuracy: 60.0 -> 62.7 | Helped |
| use 3DI-Pb-OI1M for Stage 1 alignment (ablation) | CLIP + DINOv2, Llama3-8B, Stage 2 3DI-Ft1M · SpatialVQA | Avg. accuracy: 49.0 -> 60.0 | Helped |
| use 3DI-Ft1M instead of 3DI-Pb-IN166K for Stage 1 alignment (ablation) | CLIP + DINOv2, Llama3-8B, Stage 2 3DI-Ft1M · SpatialVQA | Avg. accuracy: 62.7 -> 57.5 | Hurt |
| pretrain the vision encoder on 3DI-Pb-OI1M with CLIP objective (ablation) | CLIP + DINOv2, Llama3-8B, Stage 1 3DI-Pb-IN166K, Stage 2 3DI-Ft1M · SpatialVQA | Avg. accuracy: 62.7 -> 60.3 | Hurt |
| pretrain the vision encoder on 3DI-Pb-OI1M with NTP objective (ablation) | CLIP + DINOv2, Llama3-8B, Stage 1 3DI-Pb-IN166K, Stage 2 3DI-Ft1M · SpatialVQA | Avg. accuracy: 62.7 -> 62.6 | Hurt |
| replace DINOv2 with MAE in the mixed vision encoder (ablation) | LLaVA-v1.5 with Vicuna-7B · SpatialVQA | Avg. accuracy: 47.7 -> 47.9 | Helped |
Surprising findings
- Stage 0 3D-informed pretraining of the CLIP encoder with LoRA reduced performance, suggesting tuning visual foundation models may hurt feature generalizability.
- SpatialVLM, despite outperforming other open-source models overall, scored lower than LLaVA on 3D orientation reasoning.
Limitations
- Evaluation limited to SpatialVQA, a 1,323-question benchmark generated by rules from Omni3D annotations.
- Excludes models requiring extra inputs such as boxes, masks or depth (e.g., SpatialRGPT) from direct comparison.
- 3D orientation estimates for OpenImages probing data come from an automatic pose estimator, which is less accurate than human annotation.
Experimental setup
Datasets: 3DI-Pb-OI1M 3DI-Pb-IN166K 3DI-Ft1M CC558K Mix665K ImageNet3D OpenImages Omni3D
Benchmarks: SpatialVQA
Metrics: accuracy
Baselines: GPT-4o GPT-4o Mini Claude 3.5 Sonnet LLaVA-v1.5-7B LLaVA-NeXT-8B Cambrian-8B SpatialVLM-13B
Builds on: LLaVA-v1.5 SpatialVLM Probe3D ImageNet3D
Closest papers by meaning
- Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Mutimodal Models (CVPR 2025 · similarity 0.91)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.91)
- Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views (CVPR 2026 · similarity 0.91)
- SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.91)
- Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.90)
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.