Research map / Vision-language and multimodal / Multimodal LLMs
SpatialCLIP: Learning 3D-aware Image Representations from Spatially Discriminative Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Enhances CLIP's spatial understanding by lifting ViT tokens into 3D with depth (3D conv + 3D relative position bias) and training with recaptioned images plus spatially perturbed hard negative captions, and applies it to LLaVA-1.5 (SpatialLLaVA) with a new SpatialBench.
Problem
CLIP struggles to distinguish correct from incorrect spatial relationship descriptions in images, limiting spatial intelligence of CLIP-based systems and MLLMs.
Approach
Treat RGB-D tokens as a sparse 3D voxel grid inside a pre-trained ViT, adding 3D depth-wise convolution and 3D relative position bias, and supervise with recaptions whose spatial phrases are antonym-perturbed to serve as hard negatives.
Key results
- SpatialCLIP-B@224px raises SpatialBench outdoor/indoor/spatial average from 35.22 to 41.51 over OpenAI CLIP-B, and COCO R@1 from 51.05 to 57.34.
- SpatialCLIP-L@336px improves SpatialBench spatial score from 60.95 to 67.04 and COCO R@1 from 47.55 to 57.34 over OpenAI CLIP-L.
- SpatialLLaVA-7B reaches 40.25 outdoor and 40.17 indoor on SpatialBench versus 29.55 and 32.47 for LLaVA-1.5-7B, while keeping comparable general benchmark scores (e.g., VQAv2 79.9 vs 78.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CLIP-B with SpatialCLIP-B | OpenAI CLIP-B@224px · SpatialBench Outdoor | accuracy: 35.22 -> 41.51 | Helped |
| replace CLIP-B with SpatialCLIP-B | OpenAI CLIP-B@224px · BLINK Spatial | accuracy: 51.05 -> 57.34 | Helped |
| replace CLIP-L with SpatialCLIP-L | OpenAI CLIP-L@336px · BLINK Spatial | accuracy: 47.55 -> 57.34 | Helped |
| replace LLaVA-1.5 visual encoder with SpatialCLIP | LLaVA-1.5-7B · SpatialBench Outdoor | accuracy: 29.55 -> 40.25 | Helped |
| replace LLaVA-1.5 visual encoder with SpatialCLIP | LLaVA-1.5-7B · GQA | score: 62.0 -> 60.6 | Hurt |
| fine-tune on recaption data without 3D modules or perturbation (ablation) | OpenAI CLIP-B · SpatialBench Outdoor | accuracy: 35.22 -> 35.85 | Helped |
| add 3D CNN to recaption fine-tuning (ablation) | SpatialCLIP-B without 3D RPE or perturbation · SpatialBench Outdoor | accuracy: 35.85 -> 38.36 | Helped |
| add 3D relative position encoding to recaption fine-tuning (ablation) | SpatialCLIP-B without 3D CNN or perturbation · SpatialBench Outdoor | accuracy: 35.85 -> 37.10 | Helped |
| add 3D CNN and 3D relative position encoding (ablation) | SpatialCLIP-B with recaption fine-tuning · SpatialBench Outdoor | accuracy: 35.85 -> 38.36 | Helped |
| add spatial caption perturbation (ablation) | SpatialCLIP-B with 3D CNN and 3D relative position encoding · SpatialBench Outdoor | accuracy: 38.36 -> 41.51 | Helped |
| use early depth fusion instead of the standard CLIP setup (ablation) | OpenAI CLIP-B · SpatialBench Outdoor | accuracy: 35.22 -> 39.62 | Helped |
| use late depth fusion instead of the standard CLIP setup (ablation) | OpenAI CLIP-B · SpatialBench Outdoor | accuracy: 35.22 -> 40.25 | Helped |
Surprising findings
- Spatial benchmark performance for both CLIP and SpatialCLIP does not correlate with model size; B-sized SpatialCLIP beats L-sized OpenAI CLIP on SpatialBench spatial score (41.51 vs 32.07 outdoor).
- Early and late depth-fusion schemes often fail to improve or even hurt performance relative to the recaption-only baseline.
Limitations
- SpatialBench is small (159 outdoor and 140 indoor images) and uses caption-matching with manually crafted distractors.
- Spatial-focused performance does not scale with model size, suggesting the approach relies on added designs rather than scaling.
- Depends on monocular depth estimates from Depth Anything v2 for lifting tokens into 3D.
Experimental setup
Datasets: SA-1B COCO ShareGPT4V CoCa captions ScanNet Conceptual-12M LAION
Benchmarks: SpatialBench BLINK COCO Flickr30K POPE MM-Vet VQAv2 GQA RealWorldQA CV-Bench3D
Metrics: accuracy recall@1 recall@5
Baselines: OpenAI CLIP AlphaCLIP SpatialRGPT LLaVA-1.5 LLaVA-1.6 LLaVA-Interleave
Builds on: CLIP LLaVA-1.5 Depth Anything v2 AlphaCLIP SpatialRGPT
Closest papers by meaning
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.91)
- Refining CLIP's Spatial Awareness: A Visual-Centric Perspective (ICLR 2025 · similarity 0.91)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
- Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding (NeurIPS 2025 · similarity 0.90)
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.