Research map / Vision-language and multimodal / Multimodal LLMs
Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Spatial-MLLM adds a VGGT-initialized 3D spatial encoder alongside a Qwen2.5-VL 2D encoder, fuses them into visual tokens, selects frames via voxel-coverage greedy sampling, and trains with SFT plus GRPO to improve visual-only spatial reasoning.
Problem
Video MLLMs built on CLIP-style semantic encoders lack spatial awareness, so they perform poorly on 3D spatial reasoning when only 2D video (no depth, point clouds, or camera poses) is available.
Approach
Use a dual-encoder design that pairs the Qwen2.5-VL 2D encoder with a 3D spatial encoder initialized from the VGGT backbone, fused by an MLP connector, plus an inference-time space-aware frame sampler that greedily maximizes voxel coverage of VGGT-reconstructed geometry.
Key results
- On VSI-Bench, Spatial-MLLM-4B reaches 48.4 average (micro) with 16 frames, ranking first and above Gemini-1.5 Pro (45.4) and Qwen2.5-VL-72B (37.0).
- On SQA3D test, Spatial-MLLM-4B gets 55.9 EM-1 and 58.7 EM-R1, outperforming all video-input models.
- On ScanQA val, Spatial-MLLM-4B achieves 91.8 CIDEr versus 66.9 for Qwen2.5-VL-72B; ablation shows space-aware sampling adds about 1.3 points on VSI-Bench average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add space-aware frame sampling (ablation) | Spatial-MLLM · VSI-Bench | Avg.: 47.1 -> 48.4 | Helped |
| add GRPO training (ablation) | Spatial-MLLM after SFT · VSI-Bench | Avg.: 46.1 -> 47.1 | Helped |
| use the Spatial-MLLM architecture and training dataset rather than Qwen2.5-VL-3B SFT with question token masking (ablation) | Qwen2.5-VL-3B · VSI-Bench | Avg.: 44.9 -> 48.4 | Helped |
| SFT on the constructed spatial QA dataset (ablation) | Qwen2.5-VL-3B · VSI-Bench | Avg.: 30.6 -> 44.9 | Helped |
| increase input frames from 8 to 16 (ablation) | Spatial-MLLM · VSI-Bench | Avg.: 46.1 -> 48.4 | Helped |
| increase input frames from 16 to 32 (ablation) | Spatial-MLLM · VSI-Bench | Avg.: 48.4 -> 49.3 | Helped |
| replace uniform frame sampling with space-aware frame sampling (ablation) | Spatial-MLLM, 8 frames · VSI-Bench | Avg.: 43.8 -> 46.1 | Helped |
| replace uniform frame sampling with space-aware frame sampling (ablation) | Spatial-MLLM, 16 frames · VSI-Bench | Avg.: 47.1 -> 48.4 | Helped |
| replace uniform frame sampling with space-aware frame sampling (ablation) | Spatial-MLLM, 32 frames · VSI-Bench | Avg.: 48.4 -> 49.3 | Helped |
Surprising findings
- A 4.9B model with 16 frames outperforms 72B video MLLMs and Gemini-1.5 Pro sampled at 1 FPS on VSI-Bench.
- Even a small (1,000-step) GRPO run gives measurable gains on top of SFT.
Limitations
- Limited model scale and training data, as stated by the authors.
- Evaluated only on spatial benchmarks; effect on general video understanding not tested.
- Relies on RL training run for only 1,000 steps due to compute limits.
Experimental setup
Datasets: ScanQA SQA3D ScanNet VSI-Bench
Benchmarks: VSI-Bench ScanQA SQA3D
Metrics: mean relative accuracy accuracy BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr EM-1 EM-R1
Baselines: GPT-4o Gemini-1.5 Pro LLaVA-Video-72B LLaVA-OneVision-72B Qwen2.5-VL-72B Video-3D LLM Chat-Scene 3D-LLaVA
Builds on: Qwen2.5-VL VGGT GRPO R1-V Video-R1
Closest papers by meaning
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.93)
- Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs (NeurIPS 2025 · similarity 0.93)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.93)
- SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.93)
- Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.92)
- MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence (ICML 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.