Research map / Vision-language and multimodal / Multimodal LLMs

QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces QuantiPhy, a benchmark of ~3.3K video-text instances that quantitatively evaluates VLMs' estimation of object size, velocity, and acceleration given one physical prior, scored with Mean Relative Accuracy across 21 models.

Problem

Existing VLM physical reasoning benchmarks are qualitative and VQA/multiple-choice based, so they cannot measure whether models infer numerically accurate kinematic quantities (size, velocity, acceleration) from video.

Approach

Reframe physical reasoning as kinematic inference: given one physical prior (size, velocity, or acceleration) in real-world units plus a fixed-camera video, the VLM must output another kinematic quantity numerically, scored with relative-error-based MRA; counterfactual priors and prior-only ablations probe whether the model uses inputs or memorized world knowledge.

Key results

• ChatGPT-5.1 achieves the top overall MRA of 53.1, versus a human baseline of 55.6 and the best open-weight model Qwen3-VL-Instruct-32B at 46.0.
• Under counterfactual priors scaled by factors from 0.001 to 700, most models' MRA drops by about 80% and the strongest by about 70%, indicating reliance on memorized priors.
• Prior-only (video removed) MRA is often close to video+prior MRA, suggesting models barely use the video.
• Chain-of-thought decomposition hurts 19 of 21 models relative to direct prompting.

Surprising findings

• Prior-only (no video) performance is often close to video+prior performance, indicating models guess from textual priors.
• Step-wise chain-of-thought prompting degrades performance for 19 of 21 models.

Limitations

• Covers only translational motion with a fixed camera and rigid objects; rotation and deformable objects are excluded.
• Scenes are relatively simple, mostly isolated single or few-object motions.
• Some model APIs are unstable, and failed outputs are scored as zero.

Experimental setup

Datasets: QuantiPhy

Benchmarks: QuantiPhy

Metrics: Mean Relative Accuracy (MRA)

Baselines: Human baseline Gemini-2.5 Pro Gemini-2.5 Flash Grok 4.1 Claude Sonnet 4.5 Qwen3-VL-Instruct-32B InternVL-3.5-30B

Builds on: VSI-Bench

Closest papers by meaning

• PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.92)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.91)
• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.90)
• PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.90)
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.89)
• Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.89)