Research map / Vision-language and multimodal / Multimodal LLMs

Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds MultiSPA, a 27M-sample multi-frame spatial QA dataset from ScanNet, ADT, PStudio and TAPVid3D, and fine-tunes InternVL2-8B with LoRA into Multi-SpatialMLLM, which beats proprietary MLLMs on multi-frame spatial benchmarks.

Problem

Current MLLMs have limited spatial understanding and are largely restricted to single images, so they cannot reason about depth, correspondence, and motion across multiple frames.

Approach

A geometry-driven data engine that uses ScanNet and 4D tracking annotations to create aligned multi-frame spatial QA pairs (depth, visual correspondence, camera and object motion, object size) with dot, coordinate, and semantic referencing, used to fine-tune an MLLM while preserving its architecture.

Key results

• Multi-SpatialMLLM averages 56.11 on MultiSPA versus 20.43 for InternVL-8B and 30.31 for Gemini-2.0, a roughly 36-point gain over the base model.
• On BLINK, Multi-SpatialMLLM reaches 84.3 average versus 75.7 for Gemini-2.0 and 57.9 for InternVL-8B, with 94.7 on multi-view reasoning.
• The 26B variant reaches about 72% accuracy on camera vector prediction, reported as on par with VGGT (86 vs 82 in the expert comparison table).

Surprising findings

• Training on hard multiple-choice visual correspondence samples only improved the 26B variant; the 8B and 13B models failed to learn from them, suggesting an emergent capacity threshold.
• Adding 400K samples from unrelated tasks and datasets raised object movement accuracy from 17.5% to 22.04%, indicating cross-task transfer.

Limitations

• Data covers only two-frame scenarios; generalization to more frames is shown only for camera direction.
• Emergent behavior is preliminary and its driving abilities are not identified.
• Gains are limited on tasks like BLINK S.R. and some standard VQA benchmarks show slight drops.

Experimental setup

Datasets: MultiSPA ScanNet Aria Digital Twin Panoptic Studio TAPVid3D general image instruction data

Benchmarks: MultiSPA BLINK POPE VizWiz OCRVQA MathVista MMStar CCBench CVBench-3D ERQA MMMU MME

Metrics: accuracy exact match tolerance-based accuracy (20% relative error for scalar/vect

Baselines: InternVL2-8B InternVL2-26B SpatialRGPT Claude-3.5-Sonnet Gemini-2.0-Flash GPT-4o

Builds on: InternVL2 LoRA SpatialRGPT TAPVid3D ScanNet

Closest papers by meaning

• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.93)
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.92)
• See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.92)
• Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.92)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)