Research map / Vision-language and multimodal / Multimodal LLMs
PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PhysBench, a 10,002-entry interleaved video-image-text benchmark for VLM physical world understanding, evaluates 75 VLMs, and proposes PhysAgent (VLM plus vision foundation models and a physics knowledge memory) that improves GPT-4o zero-shot by 18.4%.
Problem
Vision-language models have limited understanding of physical world properties, relationships, scenes, and dynamics needed by embodied agents, and existing benchmarks focus on commonsense or simple synthetic primitives.
Approach
A broad multiple-choice benchmark spanning 4 physical-understanding categories and 19 sub-tasks with interleaved multi-image/video inputs, plus a training-free agent framework (PhysAgent) that lets a VLM call depth, segmentation and grounding vision models and retrieve physics knowledge from memory, keeping VLM generality rather than hand-coded pipelines.
Key results
- Across 75 VLMs, the best model GPT-4o reaches 49.49% average accuracy versus 95.87% human performance, and most models average around 40%.
- PhysAgent improves GPT-4o zero-shot performance on PhysBench by 18.4%, with a 49.5% gain in the Scene category.
- Closed-source models such as GPT-4o and Gemini-1.5 significantly outperform open-source models; GPT-4 beats the best open model LLaVA-interleave by 20.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PhysAgent to GPT-4o | GPT-4o · PhysBench | performance: 18.4% | Helped |
| Use PhysAgent instead of the base model | GPT-4o · PhysBench, physical scene understanding | performance: 49.5% | Helped |
| Use Chain of Thought prompting | GPT-4o · PhysBench | performance | No clear effect |
| Use descriptive CoT prompting | GPT-4o · PhysBench | performance | Hurt |
| Use pure language reasoning | GPT-4o · PhysBench | performance | Hurt |
| Use ContPhy instead of the base model | GPT-4o · PhysBench | performance | Hurt |
| Scale VILA-1.5 from 3B to 7B parameters (ablation) | VILA-1.5 · PhysBench | performance: decreases by 3.8% | Hurt |
| Scale VILA-1.5 from 3B to 7B parameters (ablation) | VILA-1.5 · 14 common QA tasks | performance: improves by 7.1% | Helped |
| Use larger-data variants of LLaVA-1.5 (ablation) | PLLaVA and VILA-1.5 · PhysBench | performance | Mixed |
| Increase the number of input frames (ablation) | Three open-source models · PhysBench | performance | Mixed |
| Fine-tune with physically grounded PhysBench examples | VLMs · PhysBench | performance | Helped |
| Add PhysBench data for fine-tuning | MOKA with GPT-4o · Five manipulation tasks | task performance | Helped |
Surprising findings
- Larger model size, more training data, and more frames do not reliably improve PhysBench performance, unlike common VQA benchmarks.
- ContPhy, an oracle physical reasoning method, underperforms its base model GPT-4o in three of four tasks.
Limitations
- Answer labels for the 89,998 non-test entries are withheld and test answers are hosted for online evaluation, limiting open reproduction.
- Error analysis is based on a random 500-question sample across only three models.
- PhysAgent depends on external vision tools (Depth Anything, SAM, GroundingDINO) and on GPT-4o for reasoning.
Experimental setup
Datasets: PhysBench ContPhy DROID Ego4D MimicPlay nuScenes Unsplash
Benchmarks: PhysBench
Metrics: accuracy
Baselines: Chain-of-Thought Desp-CoT Pure Language Reasoning ContPhy GPT-4o
Builds on: ContPhy MOKA Chain-of-Thought VLMEvalKit
Closest papers by meaning
- QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models (CVPR 2026 · similarity 0.92)
- PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly (NeurIPS 2025 · similarity 0.91)
- PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.90)
- PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability (CVPR 2025 · similarity 0.90)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.