Research map / Video / Video-language understanding
Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DynSuperCLEVR, a synthetic VideoQA dataset on velocity, acceleration, and collisions in 4D scenes, and NS-4DPhysics, a neural-symbolic model that parses videos into 4D scene states using a 3D neural mesh model with a physics prior and answers via program execution.
Problem
Video question answering models struggle to reason about dynamic 3D object properties (velocity, acceleration, collisions), including future prediction and counterfactual questions, because they lack explicit 3D/4D scene knowledge.
Approach
Parse each video frame with a 3D neural mesh model (analysis-by-synthesis) and a physics-engine-based prior that links consecutive frames, producing an explicit 4D symbolic scene representation; then answer factual, predictive, and counterfactual questions by executing symbolic programs, with future and counterfactual states obtained by re-simulation in a physics engine.
Key results
- NS-4DPhysics achieves 82.64% average accuracy on DynSuperCLEVR, versus 62.93% for PO3D-VQA and 59.24% for PLLaVA (best video-LLM).
- Factual accuracy of NS-4DPhysics is 87.70%, predictive 85.71%, and counterfactual 74.51%.
- Removing the physics prior drops average accuracy from 82.64% to 75.97%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the physics prior to the 4D scene representation (ablation) | NS-4DPhysics with symbolic reasoning · DynSuperCLEVR test, average | accuracy: 75.97 -> 82.64 | Helped |
| add the physics prior to the 4D scene representation (ablation) | NS-4DPhysics with symbolic reasoning · DynSuperCLEVR test, factual (All) | accuracy: 79.68 -> 87.70 | Helped |
| add the physics prior to the 4D scene representation (ablation) | NS-4DPhysics with symbolic reasoning · DynSuperCLEVR test, predictive | accuracy: 78.83 -> 85.71 | Helped |
| add the physics prior to the 4D scene representation (ablation) | NS-4DPhysics with symbolic reasoning · DynSuperCLEVR test, counterfactual | accuracy: 69.39 -> 74.51 | Helped |
| replace symbolic reasoning with GPT-4o given predicted 4D scene text (ablation) | NS-4DPhysics · DynSuperCLEVR test, average | accuracy: 82.64 -> 61.39 | Hurt |
| provide predicted 4D scene structure as text to GPT-4o instead of raw video (ablation) | GPT-4o · DynSuperCLEVR test, average | accuracy: 56.06 -> 61.39 | Helped |
| prompt GPT-4o to describe the video and reason step by step before answering | GPT-4o · DynSuperCLEVR test, average | accuracy: 51.59 -> 56.06 | Helped |
Limitations
- Only rigid objects with linear velocity and acceleration are considered; articulated and non-rigid objects are not covered.
- Synthetic dataset; real-video generalization is only shown qualitatively in an appendix case study.
Experimental setup
Datasets: SuperCLEVR Pascal3D+
Benchmarks: DynSuperCLEVR
Metrics: accuracy
Baselines: CNN+LSTM FiLM NS-DR PO3D-VQA InternVideo Video-LLaVA PLLaVA GPT-4o
Builds on: 3D neural mesh models PO3D-VQA NS-DR Kubric
Closest papers by meaning
- Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models (CVPR 2026 · similarity 0.90)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.89)
- 4DP-QA: Scalable QA for 4D Perception in Vision Language Models (CVPR 2026 · similarity 0.88)
- Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.88)
- Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.88)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.