Research map / 3D vision / 3D generation
Large Spatial Model: End-to-end Unposed Images to Semantic 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Large Spatial Model (LSM), a feed-forward Transformer that takes unposed image pairs and outputs semantic anisotropic Gaussians with geometry, appearance, and LSeg-distilled language-aligned features for real-time 3D semantic reconstruction.
Problem
Existing 3D reconstruction and open-vocabulary 3D understanding pipelines are multi-stage and depend on SfM camera poses and per-scene optimization, which is slow and fails on sparse or low-texture views.
Approach
A DUSt3R-initialized cross-view Transformer predicts pixel-aligned point maps, which a point-based Transformer (Point Transformer V3 style) refines with multi-scale cross-modal fusion of ViT and LSeg features into semantic anisotropic Gaussians rendered end-to-end, with no camera parameters needed at inference.
Key results
- Achieves 0.5078 target-view mIoU on ScanNet with 0.108s reconstruction time and no SfM or per-scene optimization, versus Feature-3DGS needing 18min36s per scene.
- Ablation: multi-scale fusion raises semantic mIoU from 0.5586 to 0.6042 over the single-fusion variant.
- Depth on ScanNet source views: relative error 3.38 and inlier ratio 67.77, better than per-scene baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal fusion of geometry encoder features (ablation) | LSM · LSeg-labeled ablation set | mIoU: 0.4562 to 0.5410 | Helped |
| add cross-modal fusion of geometry encoder features (ablation) | LSM · LSeg-labeled ablation set | Acc.: 0.6940 -> 0.8083 | Helped |
| add cross-modal fusion of geometry encoder features (ablation) | LSM · LSeg-labeled ablation set | PSNR: 24.00 -> 23.67 | Hurt |
| add cross-modal fusion of geometry encoder features (ablation) | LSM · LSeg-labeled ablation set | SSIM: 0.7981 -> 0.7876 | Hurt |
| add fusion of frozen LSeg semantic features (ablation) | LSM · LSeg-labeled ablation set | mIoU: 0.4562 to 0.5586 | Helped |
| add fusion of frozen LSeg semantic features (ablation) | LSM · LSeg-labeled ablation set | Acc.: 0.6940 -> 0.8505 | Helped |
| add fusion of frozen LSeg semantic features (ablation) | LSM · LSeg-labeled ablation set | PSNR: 24.00 -> 23.85 | Hurt |
| add fusion of frozen LSeg semantic features (ablation) | LSM · LSeg-labeled ablation set | SSIM: 0.7981 -> 0.7902 | Hurt |
| add multi-scale fusion on top of encoder-feature and LSeg-feature fusion (ablation) | LSM · LSeg-labeled ablation set | mIoU: 0.5586 to 0.6042 | Helped |
| add encoder-feature fusion, LSeg-feature fusion, and multi-scale fusion (ablation) | LSM · LSeg-labeled ablation set | Acc.: 0.6940 -> 0.8681 | Helped |
| add encoder-feature fusion, LSeg-feature fusion, and multi-scale fusion (ablation) | LSM · LSeg-labeled ablation set | PSNR: 24.00 -> 24.39 | Helped |
| add encoder-feature fusion, LSeg-feature fusion, and multi-scale fusion (ablation) | LSM · LSeg-labeled ablation set | SSIM: 0.7981 -> 0.8072 | Helped |
Surprising findings
- Target-view PSNR (24.39) is comparable to per-scene Feature-3DGS (24.49) despite being feed-forward, while the ablation shows fusing encoder features slightly lowers PSNR.
Limitations
- Relies on a pre-trained 2D model for feature lifting, which increases training GPU memory, especially for large 2D models.
- Requires ground-truth depth maps for supervision, limiting scalability to internet-scale video.
- Evaluation is limited to ScanNet and Replica with a coarse set of 8 common categories.
Experimental setup
Datasets: ScanNet++ ScanNet Replica
Benchmarks: ScanNet Replica
Metrics: mIoU Acc. rel tau PSNR SSIM LPIPS
Baselines: LSeg NeRF-DFF Feature-3DGS pixelSplat Splatter Image
Builds on: DUSt3R 3D Gaussian Splatting Point Transformer V3 LSeg pixelSplat
Closest papers by meaning
- Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning (CVPR 2024 · similarity 0.89)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.89)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.88)
- MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Image (CVPR 2025 · similarity 0.88)
- Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding (AAAI 2025 · similarity 0.88)
- G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.