Research map / Point clouds / Segmentation and detection
SpatialLM: Training Large Language Models for Structured Indoor Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes an open LLM (Qwen2.5-0.5B) with a Sonata point cloud encoder and MLP projector to output structured indoor scene layouts and oriented 3D boxes as Python-like text, trained on a new 12,328-scene synthetic dataset.
Problem
Structured indoor modeling: extracting walls, doors, windows, and oriented 3D object boxes with semantic labels from point clouds, which prior work handles with task-specific architectures and lacks large training data for.
Approach
Represent indoor layouts and object boxes as Python-style script text and fine-tune a standard "Encoder-MLP-LLM" multimodal LLM (Sonata point encoder + MLP projector + Qwen2.5) end-to-end, with a systematic study of encoder choice, token count K, and training schedule.
Key results
- On Structured3D layout estimation, SpatialLM fine-tuned from its dataset then Structured3D reaches 94.3 F1 IoU2D@0.25 and 93.5 @0.5, exceeding SceneScript (90.4/89.2) and RoomFormer (83.4/81.4).
- On ScanNet 3D detection, SpatialLM (pretrained on own dataset then fine-tuned on ScanNet) gets 65.6 F1 IoU3D@0.25, above SceneScript (49.1) and close to V-DETR (65.1), but 52.6 @0.5 vs V-DETR 56.8.
- Training on ScanNet alone yields only 2.9 F1 at IoU3D@0.25, showing small datasets are insufficient for training the LLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on the SpatialLM dataset before fine-tuning on Structured3D | SpatialLM · Structured3D | F1 IoU2D@0.25: 59.9 -> 94.3 | Helped |
| Pretrain on the SpatialLM dataset before fine-tuning on ScanNet | SpatialLM · ScanNet | F1 IoU3D@0.25: 2.9 -> 65.6 | Helped |
| Replace direct ScanNet fine-tuning with SpatialLM-dataset pretraining then ScanNet fine-tuning | SpatialLM · ScanNet | F1 IoU3D@0.5: 0.7 -> 52.6 | Helped |
| Use Sonata/PTv3 instead of the mapping-based or 3DCNN encoder alternatives (ablation) | SpatialLM encoder · Encoder design experiment | Layout F1 IoU2D@0.25: 10.8 -> 84.6 | Helped |
| Add DINOv2 features to the 3DCNN raw-point encoder (ablation) | SpatialLM encoder · Encoder design experiment | Layout F1 IoU2D@0.25: 79.4 -> 81.5 | Helped |
| Add voxelized DINOv2 features to the 3DCNN raw-point encoder (ablation) | SpatialLM encoder · Encoder design experiment | Layout F1 IoU2D@0.25: 79.4 -> 83.8 | Helped |
| Increase spatial resolution from 1× to 2× (ablation) | SpatialLM · Spatial resolution experiment | Layout F1 IoU2D@0.25: 84.6 -> 87.4 | Helped |
| Increase spatial resolution beyond 2× (ablation) | SpatialLM · Spatial resolution experiment | Layout F1 IoU2D@0.25: 87.4 -> 86.1 | Hurt |
| Train all parameters in a single stage instead of freezing the encoder (ablation) | SpatialLM · Training schedule experiment | Layout F1 IoU2D@0.25: 73.8 -> 84.6 | Helped |
| Use projector-only alignment before fine-tuning rather than single-stage training (ablation) | SpatialLM · Training schedule experiment | Layout F1 IoU2D@0.25: 84.6 -> 70.9 | Hurt |
| Use SpatialLM-dataset pretraining followed by Structured3D fine-tuning instead of either dataset alone | SpatialLM · Structured3D | F1 IoU2D@0.5: 44.7 -> 93.5 | Helped |
| Use SpatialLM-dataset pretraining followed by ScanNet fine-tuning instead of SceneScript | SpatialLM · ScanNet | F1 IoU3D@0.25: 49.1 -> 65.6 | Helped |
Surprising findings
- Training only on the small Structured3D or ScanNet datasets yields very poor results, while pretraining on the synthetic dataset first is essential.
- Increasing the point cloud token resolution beyond 2x hurt performance, attributed to excessively long token sequences.
Limitations
- Falls short of a universal model for arbitrary point clouds; fine-tuning on specific datasets is still needed for best performance.
- Limited to a predefined set of object categories, restricting open-ended language capabilities.
- Does not evaluate effects on the LLM's general language and reasoning skills.
Experimental setup
Datasets: SpatialLM dataset Structured3D ScanNet
Benchmarks: Structured3D ScanNet MASt3R-SLAM video reconstructions (zero-shot)
Metrics: F1 at IoU2D 0.25/0.5 F1 at IoU3D 0.25/0.5
Baselines: RoomFormer SceneScript V-DETR
Builds on: Sonata Point Transformer V3 SceneScript Qwen2.5 V-DETR
Closest papers by meaning
- From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.91)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.89)
- SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models (CVPR 2025 · similarity 0.89)
- Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.89)
- LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.89)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.