Research map / Vision-language and multimodal / Multimodal LLMs

S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes S²-MLLM, which fine-tunes LLaVA-Video-7B for 3D visual grounding and uses feed-forward 3D reconstruction supervision during training (not at inference) plus a structure-enhanced module with multi-level position encoding and intra/inter-view attention.

Problem

MLLMs trained on 2D image-text data struggle to understand 3D spatial structure for 3D visual grounding, and existing methods rely on explicit point-cloud reconstruction and rendering at inference, which is slow and viewpoint-dependent.

Approach

Jointly train the MLLM with a feed-forward 3D reconstruction (pointmap) objective built on Fast3R's fusion transformer, so the model internalizes 3D structure implicitly and performs latent spatial reasoning without reconstruction at inference; combine with a structure-enhanced module that adds multi-level position encoding (3D coordinates and camera ray directions) and divided intra-view/inter-vi

Key results

• Achieves 59.2% Acc@0.25 and 52.7% Acc@0.5 overall on ScanRefer, outperforming Video-3D-LLM (LoRA) at 54.1/47.9.
• On Nr3D with predicted boxes, reaches 50.6% Acc@0.25 vs. 46.1% for MCLN; on Sr3D 53.9%.
• Inference latency 1.16 s versus 1.04 s for Video-3D-LLM, with no point-cloud reconstruction at inference; SeeGround needs 3.97 s plus reconstruction time.

Surprising findings

• Removing multi-level position encoding causes the largest ablation drop (about 15 points at Acc@0.25), larger than removing spatial guidance.

Limitations

• Relies on Mask3D proposals at inference, so detection errors remain a major error category.
• Full fine-tuning of several LLM layers and attention modules is needed beyond LoRA, adding trainable parameters (about 1.77B MB reported).
• Ablation of the attention module on the full model shows little drop, so its contribution is not clearly isolated in the main table.

Experimental setup

Datasets: ScanRefer Nr3D Sr3D

Benchmarks: ScanRefer Nr3D Sr3D MultiScan ArkitScenes

Metrics: Acc@0.25 Acc@0.5 latency

Baselines: Video-3D-LLM SeeGround ZSVG3D MCLN TSP3D EDA BUTD-DETR GPT4Scene ROSS3D

Builds on: Fast3R LLaVA-Video Video-3D-LLM LoRA NeRF

Closest papers by meaning

• Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs (NeurIPS 2025 · similarity 0.92)
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.91)
• Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.91)
• Reasoning in Space via Grounding in the World (ICLR 2026 · similarity 0.91)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.91)
• SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding (NeurIPS 2025 · similarity 0.91)