Research map / Vision-language and multimodal / Multimodal LLMs

PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PhyBlock, a benchmark of 400 progressive 3D block-assembly planning tasks and 2,200 physics VQA questions, used to evaluate 21-23 VLMs on planning and physical understanding.

Problem

Vision-language models are assumed to be useful high-level planners for embodied agents, but their physical understanding and multi-step planning in structured 3D environments is poorly measured by existing benchmarks.

Approach

Uses physics-simulated 3D LEGO-like block assembly with four difficulty levels, an AOV dependency graph for partial-completion, failure-mode, and planning-robustness metrics, plus a 16-category physics VQA set, to probe VLMs on planning and physical reasoning.

Key results

• Claude-3.7 Sonnet achieves the best overall planning F1 of 47.4% (Level-1 F1 76.8%, Level-4 F1 41.8%), showing a steep drop with complexity.
• GPT-o3 achieves the highest VQA overall accuracy of 70.0%, while the random baseline is about 24.1%.
• Human experts score about 94.7% overall (378.83 of 400), setting an upper bound well above the best model.

Measured effects reported in the paper

Surprising findings

• Thinking-mode variants (Claude-3.7 Sonnet-Thinking, GPT-o1) give negligible gains over non-thinking counterparts, with orientation errors unchanged.
• Requiring absolute orientation or relative coordinates produced near-zero success, while a topology-only relaxed setting gave a substantial boost.

Limitations

• Minimalist synthetic block scenes with simple shapes and colors may not transfer to real-world visual complexity.
• Dataset is small (400 assembly tasks, 2,200 VQA) and based on manual construction and augmentation.
• Evaluation is limited to planning as a single forward pass or stepwise queries without low-level control.

Experimental setup

Datasets: PhyBlock

Benchmarks: PhyBlock

Metrics: precision recall F1 micro-F1 accuracy

Baselines: Claude-3.7 Sonnet GPT-4o GPT-o1 Gemini-2.0-flash Qwen2.5-VL-7B-Instruct InternVL2.5-78B LLaVA-OneVision-7B Random

Builds on: Genesis physics simulator Activity-on-Vertex graphs

Closest papers by meaning

• PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.91)
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.88)
• PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability (CVPR 2025 · similarity 0.88)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.88)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.88)
• Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.88)