Research map / Vision-language and multimodal / Robot action and navigation
PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Space-Physical Reachability Map (S-P Map) and PhysVLM-3B, a dual-branch VLM that encodes robot reachability as an extra image-like input, plus the Phys100K training set and EQA-phys benchmark.
Problem
VLMs often give inaccurate or impractical answers to embodied visual reasoning questions because they do not understand a robot's physical reachability, and existing representations do not generalize across robots with different kinematics.
Approach
Abstract robot reachability (from DH parameters, joint limits, calibration and depth) into a robot-agnostic S-P Map that masks unreachable image regions, and process it with a separate SigLIP feature encoder fused into a Qwen2.5-3B VLM so general vision-language ability is preserved.
Key results
- PhysVLM-3B reaches 71.0% average on EQA-phys versus 57.0% for GPT-4o, and 61.3% for GPT-4o with S-P Map.
- PhysVLM-3B obtains BLEU-4 of 43.5 on RoboVQA-val, above RoboMamba (36.3) and 3D-VLA (26.8).
- Adding the S-P Map to GPT-4o-mini raises EQA-phys average from 52.8% to 59.8% (+7.0 points); the paper's abstract reports 7.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PhysVLM-3B instead of GPT-4o | EQA-phys · EQA-phys ALL | score: 57.0 -> 71.0 | Helped |
| Add S-P Map | GPT-4o · EQA-phys ALL | score: 57.0 -> 61.3 | Helped |
| Add S-P Map | GPT-4o-mini · EQA-phys ALL | score: 52.8 -> 59.8 | Helped |
| Use PhysVLM-3B instead of GPT-4o | Robot task planning with some objects out of range · Robot task planning: part objects in range | success rate: 35.8 -> 48.4 | Helped |
| Use PhysVLM-3B instead of GPT-4o | RoboVQA-val · RoboVQA-val | BLEU-4: 36.3 -> 43.5 | Helped |
| Add S-P Map input (ablation) | PhysVLM-3B · EQA-phys REAL | score: 54.2 -> 63.5 | Helped |
| Add S-P Map input (ablation) | PhysVLM-3B · EQA-phys SIM | score: 58.8 -> 74.8 | Helped |
| Replace Depth Map with S-P Map (ablation) | PhysVLM-3B · EQA-phys REAL | score: 58.1 -> 63.5 | Helped |
| Replace Depth Map with S-P Map (ablation) | PhysVLM-3B · EQA-phys SIM | score: 62.4 -> 74.8 | Helped |
| Use an independent S-P Map feature encoder instead of sharing the visual encoder (ablation) | PhysVLM-3B · EQA-phys | score: 68.2 -> 71.0 | Helped |
| Use an independent S-P Map feature encoder instead of sharing the visual encoder (ablation) | PhysVLM-3B · OpenEQA | EM-EQA: 56.5 -> 57.4 | Helped |
| Add PyBullet data to Phys100K (ablation) | PhysVLM-3B · EQA-phys REAL | score: 62.1 -> 63.5 | Helped |
Surprising findings
- Sharing the feature encoder between the S-P Map and the image branch lowers EQA-phys (68.2 vs 71.0) and OpenEQA (56.5 vs 57.4) performance.
- PhysVLM-3B scores lower than GPT-4o on OpenEQA (57.4 vs 66.7), and is only second to GPT-4o among the compared models.
Limitations
- Zero-shot performance on real robots is lower than in simulation, likely due to domain gap.
- Phys100K reachability labels for non-PyBullet data are approximated from segmentation and depth rather than exact robot kinematics.
- Real-robot EQA-phys evaluation is small (60 samples, 300 questions).
Experimental setup
Datasets: Phys100K RoboVQA ScanNet OpenX-Embodiment LLaVA-Pretrain ShareGPT4V OpenEQA RoboVQA-val
Benchmarks: EQA-phys RoboVQA-val OpenEQA task planning (real-world pick-and-place)
Metrics: LLM score (1-5 scaled to percentage) BLEU-1/2/3/4 EM-EQA score task success rate
Baselines: GPT-4o GPT-4o-mini Claude 3.5 SpatialVLM SpatialBot 3D-VLA RoboMamba
Builds on: LLaVA SigLIP Qwen-2.5
Closest papers by meaning
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
- PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.90)
- PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies? (NeurIPS 2025 · similarity 0.89)
- QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models (CVPR 2026 · similarity 0.89)
- PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs (ICML 2024 · similarity 0.88)
- PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.