Research map / Vision-language and multimodal / Multimodal LLMs

Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes 3DThinker, which trains VLMs to produce 3D latent tokens during reasoning, aligned to VGGT features via supervised distillation and then refined with outcome-based GRPO for spatial reasoning from limited views.

Problem

VLMs struggle with 3D spatial reasoning from limited views; existing reasoning methods use pure text or 2D visual cues, or rely on auxiliary inputs, external tools, or dense cognitive-map annotations.

Approach

Insert special 3D latent tokens into the VLM's chain-of-thought, distill their last-layer hidden states (via a projector) toward a frozen 3D foundation model's (VGGT) features with an MSE/Frobenius loss, then run GRPO using only outcome rewards plus a cosine-similarity 3D reward to keep latents aligned.

Key results

• On MindCube-Tiny, 3DThinker-S1+S2 with Qwen2.5-72B reaches 78.9 overall (InternVL3-78B variant), versus 56.6 for o3; the Qwen2.5-VL-3B variant improves from 33.2 to 75.2.
• On VSI-Bench-style multi-benchmark evaluation, the Qwen2.5-VL-7B variant reaches 64.7 average vs 48.4 for VILASR-7B.
• Ablation on Qwen2.5-VL-3B shows removing the 3D alignment reward drops MindCube-Tiny from 75.2 to 68.3, and removing the answer reward drops it to 64.2.

Surprising findings

• Placing 3D latent tokens in the middle of the think block causes garbled text and a large drop (42.0 vs 75.2), while placing them at the start or end works well.
• Stage 1 alone with the 3D alignment loss removed still gives gains over CoT SFT (54.1 vs 53.4), suggesting special tokens alone add some flexibility.

Limitations

• Performance degrades with very sparse views where the 3D latent reconstruction is inaccurate, as shown in failure cases.
• Relies on VGGT features during training and GPT-4.1 to generate CoT data with 3D placeholders.
• Weaker on some sub-tasks such as Travel Time, which the authors attribute to needing richer context.

Experimental setup

Datasets: MindCube MM-Spatial SPAR SpatialLadder Ego3D-Bench

Benchmarks: MindCube-Tiny Ego3D-Bench VSI-Bench SPBench CV-Bench SPAR-Bench ViewSpatial-Bench MMSI-Bench 3DSRBench MME POPE SEED-I

Metrics: accuracy mean relative accuracy

Baselines: Ego3D-VLM MindCube cognitive-map SFT SpatialLadder-3B SpaceR-7B VILASR-7B Video-R1 Spatial-MLLM-4B SpatialReasoner SpatialThinker GRPO CoT SFT

Builds on: GRPO Mirage VGGT MindCube Ego3D-VLM

Closest papers by meaning

• Thinking with Geometry: Active Geometry Integration for Spatial Reasoning (ICML 2026 · similarity 0.93)
• Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.92)
• 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models (ICML 2026 · similarity 0.92)
• SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.92)
• Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.91)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.91)