Research map / Vision-language and multimodal / Robot action and navigation
EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EmbodiedBench, a 1,128-instance benchmark across four simulated environments (high- and low-level actions) with six capability subsets, and evaluates 24 proprietary and open-source MLLMs as vision-driven embodied agents.
Problem
Evaluation of multi-modal LLM-based embodied agents is underexplored; existing benchmarks focus on language-only agents or high-level tasks and lack fine-grained, capability-oriented assessment of low-level navigation and manipulation.
Approach
A unified benchmark spanning high-level (ALFRED, Habitat) and low-level (AI2-THOR navigation, VLMBench manipulation) embodied tasks with six capability-oriented subsets, plus a unified agent pipeline (egocentric image, in-context examples, history, environment feedback, multi-step JSON planning) and discretized manipulation actions with detection-box markers.
Key results
- GPT-4o achieves the best low-level averages, scoring 57.7% on EB-Navigation and 28.9% on EB-Manipulation.
- Claude-3.5-Sonnet leads high-level tasks with 64.0% on EB-ALFRED and 68.0% on EB-Habitat.
- Removing vision drops GPT-4o's EB-Navigation success from 57.7% to 17.4%, while high-level tasks show little dependence on vision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove vision input (ablation) | GPT-4o · EB-Navigation | success rate (Avg): 57.7% to 17.4% | Hurt |
| remove vision input (ablation) | GPT-4o · EB-ALFRED Base | success rate: 64 to 62 | No clear effect |
| remove environment feedback (ablation) | GPT-4o · EB-ALFRED Base | success rate: 10% drop | Hurt |
| remove environment feedback (ablation) | Claude-3.5-Sonnet · EB-ALFRED Base | success rate: 8% drop | Hurt |
| reduce in-context examples from 10 to 0 (ablation) | MLLM agent · EB-ALFRED Base | success rate: around 40% | Hurt |
| use 500 × 500 camera resolution instead of 300 × 300 or 700 × 700 (ablation) | MLLM agent · EB-Manipulation Base | success rate | Helped |
| add detection boxes (ablation) | GPT-4o · EB-Manipulation Base | success rate: 27.1% to 39.6% | Helped |
| add detection boxes (ablation) | Claude-3.5-Sonnet · EB-Manipulation Base | success rate: 29.2% to 37.5% | Helped |
| add images from the past two steps (ablation) | MLLM agent · EB-Manipulation Base | success rate | Hurt |
| use visual in-context examples instead of language-only examples (ablation) | Claude-3.5-Sonnet · EB-Manipulation Base | success rate: 16.7% performance boost | Helped |
Surprising findings
- Disabling vision on high-level tasks (GPT-4o Lang) matches or outperforms vision-enabled variants on EB-ALFRED and EB-Habitat.
- Current MLLMs struggle to use multiple historical image inputs, so multi-step image input degraded performance.
Limitations
- Evaluation is conducted only in simulated environments without real-world experiments.
- Multi-step/multi-image inputs are not effectively used by current MLLMs, so the agent relies mainly on the current image.
- Subset sizes are small (50 or 48 instances per subset), which limits statistical power.
Experimental setup
Datasets: ALFRED Language Rearrangement Habitat 2.0 AI2-THOR VLMBench
Benchmarks: EB-ALFRED EB-Habitat EB-Navigation EB-Manipulation
Metrics: task success rate
Baselines: GPT-4o GPT-4o-mini Claude-3.5-Sonnet Claude-3.7-Sonnet Gemini-1.5-Pro Gemini-2.0-flash Qwen-VL-Max InternVL3-78B
Builds on: ALFRED Lota-Bench Language Rearrangement VLMBench VisualAgentBench
Closest papers by meaning
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.92)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.92)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.91)
- Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.91)
- PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.91)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.