Research map / Vision-language and multimodal / Multimodal LLMs

R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Training-free R4 builds a continuous 4D (semantic, metric-spatial, temporal) object memory for VLMs and uses semantic/spatial/temporal retrieval loops to improve embodied QA and 4D reasoning.

Problem

VLMs lack persistent, queryable spatio-temporal memory for long-horizon embodied reasoning; standard RAG retrieves static text and does not index by space and time.

Approach

Anchor object-level semantic descriptions in a SLAM-based metric map and time intervals, then decompose natural-language queries into semantic, spatial, and temporal keys for iterative retrieval-augmented reasoning, shareable across agents, without retraining the VLM.

Key results

• On ERQA, R4 reaches 70.25% accuracy, above GPT-5 (65.7) and o3 (64.0) with a 4B backbone.
• On OpenEQA EM-EQA, R4 scores 79.77 LLM-Match overall, a gain of about +15.37 over the prior best reported (GPT-4V 49.6 overall per the table's ranking, with 3D-Mem at 57.2).
• On VLM4D cross-conditioned evaluation, R4 reaches 77.31 overall accuracy versus 62.0 for the best baseline (Gemini-2.5-Pro).

Surprising findings

• Temporal and spatial cues without semantic grounding (A7) give limited benefit, indicating semantics act as the essential anchor.
• Single-key retrieval yields only modest gains (max +6.4%) over the baseline on OpenEQA ablation.

Limitations

• Storage and retrieval latency limits real-time or multi-agent deployment.
• Weak on intrinsic object motion/orientation reasoning (ERQA 'other' category 42.86%).
• Multi-agent collaboration not evaluated on dedicated benchmarks; uses a curated 184-question subset for collaborative tests.

Experimental setup

Datasets: ERQA OpenEQA VLM4D nuScenes

Benchmarks: ERQA OpenEQA VLM4D

Metrics: accuracy LLM-Match LLM-Match SPL

Baselines: GPT-4V GPT-5 o3 Gemini 2.0 Pro Experimental Qwen3 VL 235B A22B Thinking 3D-Mem ConceptGraphs Sparse Voxel Map GR-ER1.5 Gemini 2.5 Pro

Builds on: retrieval-augmented generation SAM2 MapAnything OpenEQA

Closest papers by meaning

• MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence (ICML 2026 · similarity 0.89)
• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.89)
• Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.89)
• Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models (CVPR 2026 · similarity 0.89)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.88)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.88)