Research map / Vision-language and multimodal / Multimodal LLMs
SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSR, which converts depth maps into textual rationales distilled into latent tokens (via a Mamba-based MIDI module) that plug into VLMs, and releases the million-scale SSR-CoT dataset and SSRBench benchmark.
Problem
VLMs relying on RGB inputs lack precise spatial understanding, and existing depth or point-cloud integrations use depth superficially without leveraging it for higher-order spatial reasoning.
Approach
A Mamba-based Image-Depth Interpreter encodes image and Depth Pro depth features into latent tokens distilled from structured textual spatial rationales; these tokens are inserted into an existing VLM (Qwen2.5-VL), optionally with joint co-training, enabling plug-and-play depth reasoning.
Key results
- SSR (3B) reaches 64.7 on SpatialBench Position and 83.2 on SSRBench General Existence, versus 55.9 and 66.4 for Qwen2.5-VL-3B.
- Average improvement over Qwen2.5-VL backbones of 11.2 (3B) and 9.4 (7B) across the three average-reported spatial benchmarks.
- On SpatialBench, SSR (3B) averages 64.8 versus 51.3 for Qwen2.5-VL-3B with SFT on SSR-CoT, with 2.62 tokens per sample versus 437.28 and 0.32s versus 23.16s inference time per sample.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SSR MIDI rationale embeddings and Stage 2 co-training | Qwen2.5-VL 3B · SSRBench Spatial average | accuracy: 53.6 -> 76.1 (↑22.5) | Helped |
| Add SSR MIDI rationale embeddings and Stage 2 co-training | Qwen2.5-VL 3B · SSRBench General average | accuracy: 62.8 -> 79.3 (↑16.5) | Helped |
| Add SSR MIDI rationale embeddings and Stage 2 co-training | Qwen2.5-VL 3B · VQAv2, TextVQA, POPE, MMBench, and GQA | accuracy: +6.5, +4.3, +1.6, +2.4, +7.4 | Helped |
| Use latent SSR reasoning instead of generated textual CoT | Qwen2.5-VL 3B fine-tuned on SSR-CoT · SpatialBench | inference time per sample: 23.16s -> 0.32s | Helped |
| Use latent SSR reasoning instead of generated textual CoT | Qwen2.5-VL 3B fine-tuned on SSR-CoT · SpatialBench | accuracy: 51.3 -> 64.8 | Helped |
| Use MIDI plug-and-play without Stage 2 co-training (ablation) | Qwen2.5-VL 3B · SpatialBench and SSRBench | average performance: +4.4 and +1.6 | Helped |
| Add Stage 2 joint training after MIDI plug-and-play (ablation) | SSR 3B · SpatialBench and SSRBench | average performance: +5.7 and +18.7 | Helped |
| Add MIDI plug-and-play depth rationale tokens (ablation) | Qwen2.5-VL 3B · SpatialBench Position | accuracy: 55.9 -> 64.7 (↑8.8) | Helped |
| Add MIDI plug-and-play depth rationale tokens (ablation) | Qwen2.5-VL 3B · SpatialBench Existence | accuracy: 80.0 -> 80.0 (0.0) | No clear effect |
| Add MIDI plug-and-play depth rationale tokens (ablation) | Qwen2.5-VL 3B · SpatialBench Counting and Size | accuracy: ↑3.2; ↑5.0 | Helped |
| Add Stage 2 joint training after MIDI plug-and-play (ablation) | SSR 3B · SSRBench General Attribute | accuracy: 59.5 -> 82.1 (↑23.2) | Helped |
| Add Stage 2 joint training after MIDI plug-and-play (ablation) | SSR 3B · SSRBench Spatial Position | accuracy: 61.1 -> 83.3 (↑22.8) | Helped |
Surprising findings
- Latent rationale tokens achieve large gains with only 2.62 generated tokens per sample, far fewer than explicit CoT with SFT.
Limitations
- Study limited to the Qwen/Qwen-VL model series, so generalization to other VLMs is untested.
- Benchmark judgments rely on LLM-based evaluation with the same model family as judge (partly mitigated with GPT-4o-mini).
Experimental setup
Datasets: SSR-CoT LLaVA-CoT Visual-CoT VoCoT SpatialQA LLaVA-Instruct-150K
Benchmarks: SpatialBench SSRBench CV-Bench VSR What's Up VQAv2 TextVQA POPE MMBench GQA
Metrics: accuracy score (0-5) latency
Baselines: GPT-4o-mini Claude-3.5-Haiku SpatialVLM LLaVA-1.5 LLaVA-NeXT SpatialBot Emu3 Qwen2.5-VL
Builds on: Qwen2.5-VL Mamba Depth Pro SpatialRGPT Visual-CoT VoCoT LLaVA-CoT
Closest papers by meaning
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.91)
- SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.91)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.