Research map / Vision-language and multimodal / Robot action and navigation
DeepPhy: Benchmarking Agentic VLMs on Physical Reasoning
Xinrun Xu, Pi Bu, Ye Wang, Börje F. Karlsson, Ziming Wang, Tengtao Song, Qi Zhu, Jun Song, Zhiming Ding, Bo Zheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WM prompt (predict outcome) instead of VLA prompt (ablation) | agentic VLMs · PHYRE / I-PHYRE | success rate: PHYRE Claude 4.0 Opus 7.16 -> 13.33; PHYRE GPT-o3 23.06 -> 21.40; I-PHYRE GPT-o3 81.67 -> 76.67 | Mixed |
| use WM prompt instead of VLA prompt (ablation) | agentic VLMs · Kinetix (Medium/Large) | success rate: Claude 4O Medium w/: 34.72 -> 16.67; Gemini-2.5P Medium w/: 37.50 -> 34.17; GPT-o3 Medium w/o: 36.46 -> 50.00 | Mixed |
| add visual annotations of controllable components (ablation) | agentic VLMs · Kinetix Small level | success rate: GPT-o3 50.00 -> 63.33; Claude 4O 54.29 -> 68.33 | Improved |
| add visual annotations of controllable components (ablation) | agentic VLMs · Kinetix Medium/Large levels | success rate: Gemini-2.5P Medium 39.58 -> 37.50; GPT-o3 Medium 36.46 -> 40.28 | No clear effect |
| allow more attempts (attempt 1 to attempt 10) (ablation) | best VLM (GPT-o3) · PHYRE | success rate (%): 2.87 -> 23.06 | Improved |
| use frontier closed-source VLM instead of open-source Qwen | GPT-o3 vs Qwen-72B · I-PHYRE attempt 10 | success rate (%): 45.00 -> 81.67 | Improved |
| use Qwen open-source VLMs instead of random (MOCK) actions | Qwen-72B · I-PHYRE attempt 10 | success rate (%): 62.50 -> 45.00 | Hurt |
| increase Kinetix difficulty from Small to Large | GPT-o3 (VLA, w/o) · Kinetix | success rate (%): 50.00 -> 10.00 | Hurt |
Closest papers by meaning
- QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models (CVPR 2026 · similarity 0.90)
- PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.90)
- VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.89)
- Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.89)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.88)
- VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
