Research map / Vision-language and multimodal / Robot action and navigation
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
Yulin Luo, Chun-Kai Fan, Menghang Dong, Jiayu Shi, Xiangju Mi, Mengdi Zhao, Bo-Wen Zhang, Cheng Chi, Jiaming Liu, Gaole Dai, Rongyu Zhang, Ruichuan An and 10 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove image input (text-only GPT-5.4) (ablation) | GPT-5.4 · RoboBench Perception Reasoning | avg accuracy: 56.20 -> 27.12 | Hurt |
| remove image input (text-only GPT-5.4) (ablation) | GPT-5.4 · RoboBench Affordance Prediction | avg accuracy: 46.43 -> 25.61 | Hurt |
| implicit instead of explicit instruction | Claude-Sonnet-4.6 · RoboBench Instruction Comprehension | accuracy: 79.94 -> 61.38 | Hurt |
| add chain-of-thought intent scaffold to implicit instructions (ablation) | GPT-5 · Implicit plan items | world-simulator composite: 43.6 -> 45.2 | No clear effect |
| add chain-of-thought intent scaffold to implicit instructions (ablation) | Claude-Opus-4.7 · Implicit plan items | world-simulator composite: 34.2 -> 34.2 | No clear effect |
| add chain-of-thought intent scaffold to implicit instructions (ablation) | Gemini-3.1-Pro · Implicit plan items | world-simulator composite: 39.5 -> 36.4 | Hurt |
| add chain-of-thought scaffold, step verification (ablation) | GPT-5 · Implicit step-verification items | accuracy: 53.8 -> 59.0 | Improved |
| add chain-of-thought scaffold, step verification (ablation) | Claude-Opus-4.7 · Implicit step-verification items | accuracy: 64.1 -> 59.0 | Hurt |
| add chain-of-thought scaffold, step verification (ablation) | Gemini-3.1-Pro · Implicit step-verification items | accuracy: 56.4 -> 61.5 | Improved |
| use DAG-rollout prompt v3 vs basic judge v1 (ablation) | MLLM-as-world-simulator · Q1 judge consistency (200 items) | node ICC(2,k): 0.935 -> 0.948 | Improved |
| MLLM-as-world-simulator vs baseline LLM scoring | evaluation framework · human alignment | Pearson r: 0.73 -> 0.83 | Improved |
| multi-view input when front view occluded | MLLMs · Cross-View Planning | planning score | Improved |
Closest papers by meaning
- Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.91)
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.91)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.91)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.91)
- RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete (CVPR 2025 · similarity 0.91)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
