Research map / Vision-language and multimodal / Multimodal LLMs

Vision Language Models Cannot Reason About Physical Transformation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ConservationBench, a video-based benchmark of conservation tasks with paired non-conserving controls, and evaluates 112 VLMs on 23,040 trials, finding near-chance performance driven by textual invariance priors.

Problem

Whether vision-language models genuinely understand physical transformations, i.e., whether they recognize which physical quantities (number, length, volume, size) remain invariant under visual transformations versus change.

Approach

Pairs each conservation video with a matched non-conserving counterfactual (same environment, quantity altered) so that default invariance answers are penalized; varies frame count, extraction method and prompt type in a factorial design and reports strict pairwise accuracy plus textual-prior controls (empty image, text-only).

Key results

• Across 112 VLMs, average accuracy ranges from about 20% to 69% (best gemini-2.5-pro at 69.11%), versus 98.35% human baseline; 82/112 models score under 10% strict pairwise accuracy.
• Conservation accuracy and non-conserving accuracy are negatively correlated (r = -0.510, n = 112).
• Removing visual content shifts models toward 'Conserve' about 70% of the time, and model scale explains little of conservation accuracy (R^2 = 0.019).

Surprising findings

• Models perform worse on conservation tasks with real visual content (~60%) than with empty white images (85.7%), suggesting visual content disrupts the invariance prior.
• Chain-of-thought prompting significantly worsens Number & Length performance, and more frames do not reliably help.

Limitations

• Evaluation limited to four quantitative properties under controlled laboratory conditions with no occlusions or deformable objects.
• Mechanistic analysis is preliminary and conducted on a single model (Qwen2.5-VL-7B-Instruct).
• Downstream impact on planning, tool use, or robotic manipulation is untested.

Experimental setup

Datasets: ConservationBench

Benchmarks: ConservationBench BLINK MLVU MMBench-Video MME-RealWorld MMStar NaturalBench PhysBench RealWorldQA Video-MME

Metrics: accuracy strict pairwise accuracy Pearson correlation repeated-measures ANOVA

Baselines: Human Baseline gemini-2.5-pro gpt-5 claude-sonnet-4-5 doubao-seed-1.6-vision Cosmos Reason

Builds on: SeViLA VLMEvalKit

Closest papers by meaning

• PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.89)
• QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models (CVPR 2026 · similarity 0.87)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.87)
• Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.87)
• Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.87)
• Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models (ICLR 2026 · similarity 0.86)