Research map / Multi-agent systems and games / LLM agents
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VisualAgentBench, a five-environment benchmark for evaluating and behavior-cloning large multimodal models as visual agents, with a 4,482-trajectory training set built from program solvers, LMM bootstrapping, and human demos.
Problem
Existing benchmarks evaluate LMMs as agents only in single narrow environments or via offline evaluation, and provide little training data for improving open LMMs as visual agents.
Approach
A multi-domain interactive benchmark (embodied, GUI, visual design) with vision-centric observations and high-level action interfaces, paired with a hybrid trajectory curation pipeline (program-based solvers, LMM bootstrapping, human demonstrations) used for behavior cloning of open LMMs.
Key results
- gpt-4o achieves the highest average success rate of 36.2% across five environments with prompting only.
- InternVL-2 fine-tuned via behavior cloning reaches 16.0% average, outperforming gemini-1.0-pro (6.3%) and qwen-vl-max (2.7%) on all environments.
- Removing natural-language difference descriptions drops gpt-4o on VAB-CSS from 34.5% to 24.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the natural-language description of visual differences (ablation) | CSS visual-design agent · CSS | success rate: 34.5 -> 24.2 | Hurt |
| remove the natural-language description of visual differences (ablation) | CSS visual-design agent · CSS | success rate: 27.9 -> 1.9 | Hurt |
| remove the natural-language description of visual differences (ablation) | CSS visual-design agent · CSS | success rate: 29.1 -> 2.4 | Hurt |
| remove the Thought step and output actions directly (ablation) | gpt-4o agent · Minecraft | success rate: 55.2 -> 48.3 | Hurt |
| remove the Thought step and output actions directly (ablation) | gpt-4o agent · Mobile | success rate: 30.4 -> 31.9 | Helped |
| remove the Thought step and output actions directly (ablation) | gpt-4o agent · CSS | success rate: 34.5 -> 38.2 | Helped |
| remove the Thought step and output actions directly (ablation) | claude-3.5-sonnet agent · Minecraft | success rate: 56.0 -> 55.2 | Hurt |
| remove the Thought step and output actions directly (ablation) | claude-3.5-sonnet agent · Mobile | success rate: 29.0 -> 31.1 | Helped |
| remove the Thought step and output actions directly (ablation) | claude-3.5-sonnet agent · CSS | success rate: 15.8 -> 17.6 | Helped |
| add object labels to the visual input (ablation) | LMM agents in OmniGibson · OmniGibson | success rate | Helped |
| add Set-of-Marks visual annotations (ablation) | CogVLM2 GUI agent · GUI tasks | visual grounding performance | Helped |
| train open LMMs on VisualAgentBench behavior-cloning trajectories | open LMM agents · VisualAgentBench | success rate | Helped |
Surprising findings
- Directly outputting an action without a thought step performs comparably to ReAct for gpt-4o and claude-3.5-sonnet.
- Open LMMs rarely follow the system prompt without fine-tuning, yielding 0% success before training.
Limitations
- Open LMMs were only fine-tuned with smaller backbones; larger LLM backbones were not tested due to compute limits.
- Behavior cloning on offline trajectories is preliminary; no RL in diverse interactive environments.
- CSS task uses a lenient setting with natural-language difference descriptions.
Experimental setup
Datasets: WebArena Behavior-1K (OmniGibson) JARVIS-1 tasks MineRL/Minecraft Android Virtual Device apps
Benchmarks: VAB-OmniGibson VAB-Minecraft VAB-Mobile VAB-WebArena-Lite VAB-CSS
Metrics: success rate
Baselines: gpt-4o-2024-05-13 gpt-4-vision-preview gpt-4-turbo-0409 claude-3.5-sonnet claude-3-opus gpt-4o-mini gemini-1.5-pro gemini-1.0-pro qwen-vl-max InternVL-2 GLM-4V LLaVA-NeXT
Builds on: ReAct Steve-1 JARVIS-1 WebArena Behavior-1K
Closest papers by meaning
- AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.91)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents (NeurIPS 2026 · similarity 0.89)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.89)
- Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (CVPR 2024 · similarity 0.89)
- DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.