Research map / Vision-language and multimodal / Multimodal LLMs

MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds an automated pipeline that turns stereo videos (Stereo4D) into 4D spatiotemporal QA data, then trains Qwen-VL models with LoRA SFT, a cold start, and GRPO with spatiotemporal (camera/object center) rewards to improve visual 4D understanding and reasoning.

Problem

MLLMs struggle to understand and reason about the evolution of 3D space over time from purely 2D video, and existing 4D instruction data and benchmarks are small and manually annotated.

Approach

Repurpose stereo video datasets with camera poses, metric 3D points and segmentation to auto-generate physics-computed 4D QA pairs, then post-train a standard (encoder-free-addition) MLLM with SFT followed by GRPO using ST-CoT prompting and a spatiotemporal reward on camera and object center coordinates.

Key results

• MLLM-4D (Qwen3-VL-8B) reaches 72.7% average on MLLM4D-Bench, vs 46.6% for Gemini2.5-Pro and 35.3% for the Qwen3-VL-8B baseline.
• On VLM4D real-world videos, MLLM-4D (Qwen3-VL-8B) scores 63.1% overall vs 52.2% for Qwen3-VL-8B baseline (and 62.0% for Gemini-2.5-Pro overall in the table).
• Ablation: GRPO with ST-reward improves MLLM4D-Bench from 70.5% (without ST-reward) to 72.7%, and SFT with the curated data lifts it from 59.9% to 70.1%.

Surprising findings

• Proprietary Gemini2.5-Pro scores 62.0% overall on VLM4D, above the paper's own MLLM-4D (Qwen3-VL-8B) 61.0% in the overall column of that table, though MLLM-4D is higher on the real-split comparison.
• Monocular-video-based alternative pipeline is reported to yield weaker gains than the stereo pipeline, attributed to depth ambiguity.

Limitations

• Relies on frame sampling (32 frames) due to input length limits; no long-context 4D reasoning.
• Data depends on stereo video datasets and multi-stage vision pipeline (Gemini, GroundedSAM2, PixelRefer, camera pose and stereo depth) that may carry errors.
• Gains on some subtasks (e.g., Qwen2.5-VL-7B version on Object-Camera Rel. Dir.) are modest or mixed.

Experimental setup

Datasets: Stereo4D MLLM4D-2M MLLM4D-R1-30k VLM4D

Benchmarks: MLLM4D-Bench VLM4D

Metrics: accuracy average accuracy across subtasks

Baselines: GPT-4o Gemini2.5-Flash Gemini2.5-Pro Qwen2.5-VL-7B Qwen3-VL-8B VG-LLM VLM-3R InternVL3.5-8B

Builds on: GRPO LoRA Stereo4D Qwen3-VL GroundedSAM2

Closest papers by meaning

• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.92)
• Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs (NeurIPS 2025 · similarity 0.91)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.91)
• Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.91)
• Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.90)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)