Research map / Vision-language and multimodal / Multimodal LLMs

4DP-QA: Scalable QA for 4D Perception in Vision Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a scalable QA generation pipeline and a 400K-sample 4D motion QA training set (4DP-QA) plus a 2.2K benchmark, adding a novel true-motion point tracking task that disentangles camera and object motion.

Problem

VLMs struggle with 4D (3D plus time) motion understanding because motion is observed only as 2D projection on a moving camera, and existing datasets do not disentangle object motion from camera motion.

Approach

Generate large-scale templated QA pairs from geometric annotations (camera poses, depth, 6D object poses) across real and synthetic datasets, and introduce true-motion point tracking, which projects object tracks into a fixed reference camera frame so that background stays stationary and only true object motion remains.

Key results

• Qwen2.5-VL-7B rises from 46.6% to 84.3% overall on 4DP-QA-Bench after training on 4DP-QA; NVILA-Lite-8B from 42.3% to 84.4%.
• On VLM4D, NVILA-Lite-8B improves from 42.8% to 60.5% and Qwen2.5-VL-7B from 52.3% to 63.6%.
• Adding true-motion tracking data gives the best VLM4D results on average across baselines, per the ablation.

Surprising findings

• Several open-source VLMs score below random guessing on some 4DP-QA-Bench categories, showing systematic bias toward wrong answers.
• Qwen2.5-VL-3B fine-tuned on 4DP-QA matches the larger Qwen2.5-VL-32B and Qwen3-VL-32B on VLM4D.

Limitations

• Gains from the geometry encoder are mixed: VLM4D real split degrades, attributed to uniform 32-frame sampling being too sparse for L4P.
• Benchmark evaluation uses exact string matching, and absolute coordinate questions are excluded from the benchmark because coordinate conventions vary across VLMs.

Experimental setup

Datasets: SHIFT Virtual KITTI 2 Aria Digital Twin HOT3D Kubric VLM4D

Benchmarks: 4DP-QA-Bench VLM4D

Metrics: accuracy

Baselines: GPT-4o Gemini-2.5-Pro Gemini-2.5-Flash Gemini-Robotics Qwen2-VL-7B Qwen3-VL-8B Qwen2.5-VL-32B Qwen3-VL-32B LLaVA-OneVision-7B LLaVA-OneVision-1.5-8B NVILA-Video-8B

Builds on: NVILA L4P Qwen2.5-VL VLM4D

Closest papers by meaning

• Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.91)
• 4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration (NeurIPS 2025 · similarity 0.89)
• Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering (ICLR 2025 · similarity 0.88)
• QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models (CVPR 2026 · similarity 0.88)
• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
• Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.88)