Research map / Vision-language and multimodal / Multimodal LLMs

Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that VLMs linearly bind spatial (and temporal) IDs to object-word activations in intermediate layers, and that these IDs causally steer model beliefs, diagnose failures, and can serve as a auxiliary training signal.

Problem

The internal mechanism by which VLMs combine visual spatial/temporal information with textual reasoning for spatial and temporal VQA remains opaque.

Approach

Hypothesizes that spatial location is linearly bound to object-word token activations in intermediate layers ("spatial IDs"), extracts them by averaging out object semantics over a grid of synthetic renders, and validates causality via mirror swapping and residual steering; extends to temporal IDs in video VLMs.

Key results

• Spatial ID steering yields a median 64.6% binary belief swap versus 29.5% for noise steering (about 43.6% above chance on average) across 11 VLMs.
• Finetuning Qwen2-2B with a spatial ID loss reaches 91% COCO-Spatial validation accuracy at 3.2k steps versus 85% for the control.
• Rank-3 linear fit from positional encodings to spatial IDs yields R^2 of roughly 0.85 or above.

Surprising findings

• LLaVA and LLaMA show depth and height representations that are strongly correlated, with 'above'/'behind' and 'below'/'front' mapping to overlapping ID locations.
• Attribute (color) swapping produces mostly noise, with belief shifts up to 20x the original belief difference, showing spatial swapping is a cleaner probe.

Limitations

• Only models up to 14B parameters were studied due to compute constraints.
• Analysis is limited to simple spatial queries and appearance-based temporal queries.
• Diagnoses are described as preliminary and not attributable to a single architectural component.

Experimental setup

Datasets: COCO-Spatial COCO Objaverse MVBench

Benchmarks: COCO-Spatial MVBench Scene_QA

Metrics: binary belief swap rate belief shift log probability change accuracy Mann-Whitney U p-value R^2

Baselines: noise steering

Builds on: Linear Representation Hypothesis binding IDs in LLMs activation patching D-RISE

Closest papers by meaning

• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.93)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.92)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (NeurIPS 2026 · similarity 0.91)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
• Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs (NeurIPS 2025 · similarity 0.91)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)