Research map / Vision-language and multimodal / Multimodal LLMs

Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Surprise3D, a ScanNet++-based dataset of 200k+ language-query/3D-mask pairs (89k human-written spatial queries without object names plus LLM-generated common-sense/intention queries) and the 3D-SRS benchmark, showing that existing 3D grounding methods and 3D-LLMs perform poorly zero-shot and only moderately after fine-tuning.

Problem

Existing 3D vision-language grounding datasets mix object names with spatial context, letting models rely on semantic shortcuts instead of genuine spatial reasoning, and they lack fine-grained spatial reasoning coverage.

Approach

Build a large segmentation benchmark whose spatial queries deliberately omit object names and span relative position, narrative perspective, parametric perspective, and absolute distance reasoning, with human annotation for spatial queries and LLM-generated then human-verified queries for common-sense and intention reasoning.

Key results

• Zero-shot overall mIoU is at most 8.89 (MLLMfor3D), with Reason3D at 6.08 and Acc@25 of 9.09 overall, showing weak spatial reasoning of existing methods.
• After fine-tuning on Surprise3D, MLLMfor3D reaches 16.46 overall mIoU and Intent3D reaches 23.98 overall Acc@25, with spatial reasoning gains reported as about threefold on average.
• Parametric perspective is the hardest spatial category: zero-shot MLLMfor3D gets 2.93 mIoU and fine-tuned Reason3D gets 8.85.

Measured effects reported in the paper

Surprising findings

• Zero-shot models score higher on common-sense and intention queries than on spatial reasoning queries, and ChatScene scores 0 on several spatial categories.
• The authors find that LLMs and MLLMs cannot generate spatial reasoning annotations with sufficient fidelity, requiring human annotation.

Limitations

• Human annotation limits scalability.
• Some query types such as parametric perspective may be less natural for real deployment.
• Restricted to indoor ScanNet++ scenes, may not generalize to outdoor or dynamic environments.

Experimental setup

Datasets: ScanNet++ v2 ScanRefer SQA3D

Benchmarks: Surprise3D 3D-SRS

Metrics: mIoU Acc@0.25 Acc@0.5

Baselines: MLLMfor3D 3D-Vista Reason3D Intent3D ChatScene

Builds on: ScanNet++ ScanRefer ReferIt3D ScanReason

Closest papers by meaning

• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.91)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
• Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.91)
• RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics (CVPR 2025 · similarity 0.91)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
• 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models (ICML 2026 · similarity 0.91)