Research map / Vision-language and multimodal / Robot action and navigation
PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PAC Bench, a benchmark of over 30,000 annotations that evaluates whether VLMs understand object properties, affordances, and physical constraints needed to execute manipulation tasks, finding large gaps especially on affordance completeness and constraint reasoning.
Problem
Vision-language models are used for robot manipulation (policy generation, failure detection, physical reasoning), but their understanding of low-level physical prerequisites, namely object properties, action affordances, and physical constraints, has not been systematically verified.
Approach
A diagnostic benchmark that decomposes manipulation prerequisites into Properties, Affordances, and Constraints (PAC), combining web images, Unitree G1 humanoid captures, and MuJoCo simulated scenes, and evaluates pre-trained VLMs zero-shot on each dimension.
Key results
- Llama 4 Scout reaches 40.9% average on at-least-one affordance accuracy, the highest among evaluated models in the affordance table, while full-set affordance accuracy is near zero for most models.
- Constraint accuracy is very low: Claude 3.5 Sonnet averages 0.4% and Llama 3.2 90B Vision averages 0.2%, while Gemini 2.5 Pro Preview averages 25.8% across simulated constraint domains.
- Gemini 2.5 Pro Preview reaches up to 90.0% on simulated occlusion front-view, but scores vary strongly across viewpoints (30.0% agent view).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add chain-of-thought prompting (ablation) | Claude 3.7 Sonnet · PAC Bench properties (Open Images + Humanoid, Avg) | accuracy: 38.9 -> 23.1 | Hurt |
| Add chain-of-thought prompting (ablation) | Claude 3.7 Sonnet · PAC Bench affordances (at least one correct affordance, Avg) | accuracy: 24.4 -> 9.4 | Hurt |
| Add chain-of-thought prompting (ablation) | Claude 3.7 Sonnet · PAC Bench constraints (simulation + real-world, Avg) | accuracy: 5.6 -> 7.5 | Helped |
| Evaluate on humanoid robot views rather than Open Images | Qwen 3 · PAC Bench properties (six evaluated properties) | accuracy | Hurt |
| Require all ground-truth affordances rather than at least one correct affordance | VLMs evaluated on PAC Bench · PAC Bench affordances | accuracy | Hurt |
| Evaluate on real-world humanoid scenarios rather than simulated constraints | Gemini 2.5 Pro P · PAC Bench constraints | accuracy: 25.8% Avg; 11.3% Agent, 18.8% Side, 9.4% Both | Hurt |
Surprising findings
- Performance on simulated constraint scenarios does not transfer to real-world humanoid views; GPT-4.1 Mini scores near zero in simulation but 22.6% on humanoid both-views.
- Qwen 3 scores 0.0% across all six properties on the humanoid split despite reasonable Open Images performance.
Limitations
- Affordance annotation used a single primary annotator per object class, without inter-annotator agreement.
- RoboCasa subset was excluded from current VLM evaluations due to computational cost.
- Models were accessed via OpenRouter between April and May 2024, and some model versions are inconsistent with the stated dates.
Experimental setup
Datasets: OpenImages V7 RoboCasa Unitree G1 humanoid captures MuJoCo simulated constraint scenarios
Benchmarks: PAC Bench
Metrics: accuracy at-least-one affordance accuracy all-affordance accuracy
Baselines: Claude 3.5 Sonnet Claude 3.7 Sonnet Gemini 2.0 Flash Gemini 2.5 Pro GPT-4.1 GPT-4.1 Mini o4-mini-high Llama 4 Maverick Llama 4 Scout Qwen 2.5 VL Qwen 3 Grok 2 Vision
Builds on: HELM VHELM PhysBench UniAff RoboCasa
Closest papers by meaning
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.91)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.90)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.89)
- PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability (CVPR 2025 · similarity 0.89)
- PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.