Research map / Vision-language and multimodal / Multimodal LLMs

Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Dyn-Bench, a benchmark of 1k dynamic videos with 7k VQA pairs and 3k grounding annotations for evaluating MLLM spatio-temporal reasoning and dynamic object grounding, and proposes ST-TCM (textual spatio-temporal cognitive map) and mask-guided fusion as structured enhancements.

Problem

Existing MLLMs are strong on static images but unclear at perceiving, tracking, and reasoning about object-centric spatio-temporal dynamics in evolving 4D scenes; prior benchmarks focus on scene-level dynamics and lack dynamic object grounding.

Approach

A benchmark pairing multiple-choice dynamic VQA with mask grounding (J&F) across inter-object, object-scene, and camera-object levels, plus a rule-based Spatio-Temporal Textual Cognitive Map built from reconstructed 3D trajectories that is fed to MLLMs as text, and a mask-guided fusion input that overlays raw frames with masks.

Key results

• Qwen3-VL-235B achieves 65.3 average accuracy on Dyn-Bench VQA, the top open-source model, ahead of Gemini-2.5 Pro (59.8) and GPT-5 (59.5).
• Sa2VA-InternVL2.5-8B reaches 75.6 J&F average on dynamic object grounding, the highest among region-level models evaluated.
• Adding all three ST-TCM components (T+M+S) raises Qwen3-VL-32B average accuracy from 62.8 to 68.3.
• Mask-Guided Fusion improves Qwen3-VL-8B average accuracy from 53.8 to 57.1, while masked frames alone give no average gain.

Surprising findings

• Chain-of-thought and caption-based prompting yield only limited improvement on dynamic reasoning, while structured textual maps and mask-guided fusion help substantially.
• Region-level MLLMs that excel at grounding do not consistently lead on reasoning; no single model balances both well.

Limitations

• Benchmark VQA uses multiple-choice format with exact-match accuracy, which may not capture open-ended reasoning.
• ST-TCM and QA pairs are generated with Qwen3-VL-235B, so the benchmark may inherit that model's biases.

Experimental setup

Datasets: DAVIS SA-V DynPose-100K YouTube-VIS DynamicReplica PointOdyssey Spring Total-Recon

Benchmarks: Dyn-Bench

Metrics: accuracy J&F

Baselines: GPT-4o GPT-5 Gemini-2.5 Pro Qwen3-VL-235B SpaceR-7B VST-7B-RL UniPixel-7B Sa2VA-InternVL2.5-8B

Builds on: Chain-of-thought prompting Sa2VA VIPE VGGT

Closest papers by meaning

• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.91)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.90)
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (CVPR 2025 · similarity 0.90)
• CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning (NeurIPS 2026 · similarity 0.90)
• Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.90)
• MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence (ICML 2026 · similarity 0.90)