Research map / Large language models / AI and society
ContPhy: Continuum Physical Concept Learning and Reasoning from Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ContPhy, a simulated video question-answering benchmark for physical property inference and dynamics reasoning on continuum objects (fluids, ropes, cloth, soft balls), and an oracle neuro-symbolic model ContPRO that combines Mask R-CNN perception, particle-based simulators (MPM, DPI-Net), and GPT-4 program parsing.
Problem
Existing physical reasoning benchmarks focus on rigid objects with simple primitives and fixed physical parameters, and lack diverse language-based questions about physical properties and counterfactual or goal-driven dynamics for soft bodies and fluids.
Approach
A Unity-simulated benchmark with varying physical parameters (mass, density, elasticity, plasticity, etc.) across four continuum scenarios, with template-based question generation, paired with an oracle model that uses an LLM to parse questions into programs executed over particle-based simulator outputs.
Key results
- ContPRO reaches 71.4% on rope property questions and 90.1% on fluid predictive options, the best among machine models in several settings.
- Human participants (16 people, 460 answers) outperform all models in every scenario, e.g., 84.7% rope property vs 71.4% for ContPRO.
- MLLMs GPT-4V and Gemini perform poorly on fluid and cloth questions (e.g., fluid property accuracy 25.0% and 10.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add point cloud features to visual models (ablation) | CNN-LSTM and MAC on ContPhy · ContPhy physical reasoning settings | accuracy: 52.7 -> 55.0; 53.3 -> 57.7; 67.5 -> 68.3; 45.5 -> 53.4 | Helped |
| add point cloud features to MAC (ablation) | MAC on fluid predictive questions · Fluid P-Ques. | accuracy per question: 12.5 -> 13.5 | Hurt |
| use LLM question rephrasing instead of templates (ablation) | ContPhy question generation · Fluid questions | TTR: 0.0096 -> 0.052 | Helped |
| use LLM question rephrasing instead of templates (ablation) | ContPhy question generation · Rope questions | TTR: 0.0096 -> 0.053 | Helped |
| use LLM question rephrasing instead of templates (ablation) | ContPhy question generation · Cloth questions | TTR: 0.0089 -> 0.068 | Helped |
| use LLM question rephrasing instead of templates (ablation) | ContPhy question generation · Ball questions | TTR: 0.0066 -> 0.049 | Helped |
| use the NEWTON text-only prompting approach (ablation) | MLLM prompting on ContPhy · Rope property | accuracy: 35.5 -> 50.0 | Helped |
| use the NEWTON text-only prompting approach (ablation) | MLLM prompting on ContPhy · Fluid property | accuracy: 10.0 -> 4.0 | Hurt |
| use human-explained examples (ablation) | MLLM prompting on ContPhy · Rope goal-driven questions | accuracy per question: 10.3 -> 19.0 | Helped |
| use higher-resolution upsampled video (ablation) | MLLM prompting on ContPhy · Fluid counterfactual questions | accuracy per question: 6.0 -> 0.0 | Hurt |
Surprising findings
- ContPRO's accuracy on predictive questions surpasses human performance, which the authors attribute to MPM giving precise short-term predictions while humans give vague estimates.
- Blind models perform similarly to other families on some goal-driven rope questions.
Limitations
- Limited language diversity due to predefined templates and vocabulary.
- Four scenarios do not cover the complexity of real-world physical interactions.
- ContPRO requires point cloud / particle-based scene inputs, making it an oracle model with extra supervision.
Experimental setup
Datasets: ContPhy
Benchmarks: ContPhy
Metrics: accuracy per option accuracy per question
Baselines: RND FRQ B-LSTM C-LSTM HCRN MAC ALPRO Violet PhyDNet PIP Gemini GPT-4V
Builds on: ViperGPT DPI-Net Material Point Method Mask R-CNN ALPRO
Closest papers by meaning
- Scaling Physical Reasoning with the PHYSICS Dataset (NeurIPS 2025 · similarity 0.85)
- Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models (ICLR 2026 · similarity 0.85)
- Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence (NeurIPS 2026 · similarity 0.84)
- PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.84)
- QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models (CVPR 2026 · similarity 0.84)
- UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.