Research map / Vision-language and multimodal / Multimodal LLMs
Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Struct2D, a perception-guided 2D prompting framework (BEV image with object marks, metadata, optional keyframes) that enables zero-shot spatial reasoning in MLLMs, and builds the 200K-QA Struct2D-Set to fine-tune Qwen2.5-VL for 3D QA, grounding, and captioning without 3D inputs.
Problem
Spatial reasoning in MLLMs typically requires explicit 3D inputs (point clouds) or specialized architectures; the paper asks whether MLLMs can reason about 3D space from only structured 2D representations derived from 3D perception.
Approach
Convert off-the-shelf 3D perception outputs into a single BEV image with query-filtered object marks, object-centric metadata (categories, 3D coordinates), and optional egocentric keyframes selected via 3D projection, feeding these 2D inputs to MLLMs for zero-shot and fine-tuned spatial reasoning.
Key results
- Zero-shot GPT-o3 with Struct2D prompting (noisy objects) reaches 56.1 average on a 422-question VSI-Bench subset, vs 48.6 for 16-frame video input, and 94.4 on relative direction with GT objects.
- Qwen2.5-VL-7B fine-tuned on Struct2D-Set achieves 43.6 average on VSI-Bench, the highest among open-source models listed, above R1-Zero-VSI+SFT (38.8).
- On ScanQA val, Qwen2.5-VL-7B (Ours) reaches CIDEr 92.1 and EM-R1 61.3 on SQA3D, slightly above GPT4Scene-style Qwen2-VL-7B baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on Struct2D-Set instead of the standard baseline | Qwen2.5-VL-3B · VSI-Bench | Avg.: 33.9 -> 41.9 | Helped |
| replace standard baseline with zero-shot Struct2D prompting | Qwen2.5-VL-3B · VSI-Bench | Avg.: 33.9 -> 29.4 | Hurt |
| use Struct2D-Set fine-tuning rather than Struct2D prompting | Qwen2.5-VL-3B · VSI-Bench | Avg.: 29.4 -> 41.9 | Helped |
| provide Struct2D inputs with ground-truth rather than noisy objects | GPT-o3 · VSI-Bench zero-shot subset | Avg.: 56.1 -> 83.8 | Helped |
| add object-centric metadata (ablation) | GPT-o3 with filtered marks · VSI-Bench zero-shot subset | Rel. Dist.: 72.1 -> 96.5 | Helped |
| add object-centric metadata (ablation) | GPT-o3 with filtered marks · VSI-Bench zero-shot subset | Route Plan: 78.3 -> 80.1 | Helped |
| filter object marks based on question relevance (ablation) | GPT-o3 with metadata · VSI-Bench zero-shot subset | Route Plan: 50.6 -> 80.1 | Helped |
| add object-centric metadata and filtered marks (ablation) | GPT-o3 · VSI-Bench zero-shot subset | Rel. Dist.: 67.5 -> 96.5 | Helped |
| add a structured guide prompt (ablation) | GPT-o3 with rotation alignment · VSI-Bench zero-shot subset | Rel. Dir.: 89.2 -> 94.4 | Helped |
| add rotation alignment (ablation) | GPT-o3 with guide prompt · VSI-Bench zero-shot subset | Rel. Dir.: 80.2 -> 94.4 | Helped |
| use augmented QA pairs (ablation) | Qwen2.5-VL-3B · VSI-Bench | Avg.: 31.5 -> 38.0 | Helped |
| use augmented QA pairs (ablation) | Qwen2.5-VL-3B · VSI-Bench | Rel. Dir.: 14.7 -> 42.2 | Helped |
Surprising findings
- Zero-shot GPT-o3 with structured 2D prompts outperforms 16-frame video input by a wide margin, and with ground-truth objects reaches 83.8 average, suggesting perception quality is the main bottleneck.
Limitations
- Relies on 3D perception modules (reconstruction and detection) at preprocessing; noisy reconstructions cause many failures.
- Evaluated mainly on indoor scenes; generalization to outdoor or open-world scenes is unexplored.
- Rule-based metrics (BLEU, CIDEr) may not reflect reasoning quality; Struct2D-Set reported as released only in promises.
Experimental setup
Datasets: Struct2D-Set ScanNet ScanNet++ ARKitScenes ScanQA SQA3D Scan2Cap ScanRefer Multi3DRefer
Benchmarks: VSI-Bench ScanQA SQA3D ScanRefer Multi3DRefer Scan2Cap
Metrics: accuracy BLEU-4 CIDEr METEOR ROUGE EM-R1 Acc@0.25 Acc@0.50 F1@0.25 F1@0.50
Baselines: GPT4Scene R1-Zero-VSI LLaVA-OneVision-7B LLaVA-NeXT-Video-7B InternVL2-8B Chat-Scene LEO Qwen2-VL-7B (GPT4Scene)
Builds on: GPT4Scene VSI-Bench Qwen2.5-VL chain-of-thought
Closest papers by meaning
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.93)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.93)
- S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance (CVPR 2026 · similarity 0.92)
- Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.92)
- EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs (CVPR 2026 · similarity 0.92)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.