Research map / Multi-agent systems and games / LLM agents
AgentStudio: A Toolkit for Building General Virtual Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgentStudio, a lightweight online environment with generic video/image observations and GUI+API actions, plus tools, a 205-task online benchmark, and three datasets (GroundUI, IDMBench, CriticBench) for decomposing virtual-agent abilities.
Problem
Existing virtual-agent environments are domain-specific, hard to set up, lack video observations and combined GUI/API actions, and evaluations rarely decompose fundamental agent capabilities such as GUI grounding, learning from videos, and success detection.
Approach
A generic online environment whose observation space (text, image, video) and action space (GUI keyboard/mouse plus API/code) are universal, paired with authoring tools (task creation with auto-evaluators, GUI element annotation, video action labeling) and three fine-grained datasets that each isolate one agent ability.
Key results
- Claude 3.5 Sonnet and GPT-4o reach 36.6% and 35.6% total success on the 205-task online benchmark versus 72.2% for humans.
- On GroundUI-1K, SeeClick achieves 61.1% overall grounding accuracy, the best among tested models; Gemini 1.5 Pro leads proprietary models at 35.2%.
- On IDMBench IDM-Multiple, the best model (Claude 3.5 Sonnet) reaches only 12.5% accuracy versus 61.4% on IDM-Single.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use screenshot observations in addition to text (Single-GUI OS vs Single-API OS) (ablation) | GPT-4o and Claude 3.5 Sonnet · AgentStudio Single-API vs Single-GUI OS tasks | success rate (%) | No clear effect |
| Add screenshot observations (Single-GUI OS vs Single-API OS) (ablation) | Gemini models · AgentStudio Single-API vs Single-GUI OS tasks | success rate (%) | Hurt |
| Include action information alongside observations in CriticBench (ablation) | VLMs as critics · CriticBench | F1 score: limited benefit | Helped |
| Switch from single-step (IDM-Single) to multi-step action labeling (IDM-Multiple) | VLMs as IDMs · IDMBench | accuracy (%): large drop, e.g. Claude 61.4 to 12.5 | Hurt |
| Recaption ambiguous/incorrect GUI instructions with GPT-4o (ablation) | GroundUI-1K grounding evaluation · GroundUI-1K | accuracy | Mixed |
Surprising findings
- Providing screenshots can hurt API-solvable tasks because models switch to GUI actions instead of efficient API calls.
- Success detection performance is similar across open and proprietary VLMs, and action information adds only limited benefit.
Limitations
- Auto-evaluators may make incorrect judgments in real-world complexity.
- GroundUI recaptioning relies on GPT-4o and may still contain problematic instructions.
- Online benchmark results obtained in an Ubuntu Docker environment, and datasets may be vulnerable to hacks or leakage.
Experimental setup
Datasets: Mind2Web AITW VisualWebArena ScreenSpot OmniACT MoTIF
Benchmarks: AgentStudio online benchmark (205 tasks) GroundUI-1K IDMBench CriticBench
Metrics: success rate grounding accuracy accuracy F1 score edit distance
Baselines: GPT-4o Claude 3.5 Sonnet Gemini 1.5 Pro Gemini 1.5 Flash SeeClick CogAgent Qwen-VL-Chat
Builds on: WebArena OSWorld AndroidWorld Mind2Web SeeClick
Closest papers by meaning
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.88)
- AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials (ICLR 2025 · similarity 0.88)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.88)
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents (NeurIPS 2026 · similarity 0.88)
- OpenCUA: Open Foundations for Computer-Use Agents (NeurIPS 2025 · similarity 0.88)
- AgenticVBench: Can AI Agents Complete Real-World Video Production Tasks? (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.