Research map / Multi-agent systems and games / LLM agents
What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OmniBench, a self-generating graph-based benchmark of 36k controllable-complexity GUI tasks across 20 scenarios, plus OmniEval graph-based metrics (CR, LC), and evaluates 12 agents showing graph-structured tasks are hard.
Problem
Existing virtual agent benchmarks have fixed, uncontrollable task complexity, rely on manual annotation, and evaluate only final outcomes or trajectory similarity, giving little multidimensional capability feedback.
Approach
Bottom-up automatic synthesis of task DAGs from subtasks linked by input/output resources, with controllable complexity along five graph dimensions, intent extraction and consistency validation, paired with a graph-based evaluator using Coverage Rate and Logical Consistency metrics across 10 capabilities.
Key results
- GPT-4o achieves 38.7 overall CR on OmniBench versus 80.1 for humans; on graph-structured tasks GPT-4o reaches only 20.5%.
- Omni-UGround-V1-7B reaches 34.4 overall CR, the best among agents tested, vs 25.0 for UGround-V1-7B.
- The automated pipeline reaches 90.7% human acceptance with all quality modules versus 41.2% without them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add task intent to the prompt (ablation) | Closed-source planners on OmniBench · OmniBench planning | Overall planning score: GPT-4o 25.4 to 34.3; Claude-3.5-Sonnet 24.0 to 29.4; Gemini-2.0-Flash 22.9 to 27.8; Qwen-VL-Max 21.4 to 24.0 | Helped |
| Add task intent to fine-tuning data (ablation) | OS-Atlas-Base-4B and UGround-V1-7B · OmniBench planning | Overall planning score: 28.6 to 30.3; 32.3 to 33.5 | Helped |
| Remove cross-verification (ablation) | OmniBench task synthesis · Synthesized task acceptability | Human Acceptance: 90.7% to 61.2% | Hurt |
| Remove intent extraction (ablation) | OmniBench task synthesis · Synthesized task acceptability | Human Acceptance: 90.7% to 82.7% | Hurt |
| Remove consistency validator (ablation) | OmniBench task synthesis · Synthesized task acceptability | Human Acceptance: 90.7% to 86.5% | Hurt |
| Remove all quality-control modules (ablation) | OmniBench task synthesis · Synthesized task acceptability | Human Acceptance: 90.7% to 41.2% | Hurt |
| Fine-tune UGround-7B-V1 on OmniBench graph-structured trajectories | UGround-7B-V1 · AndroidControl and OmniAct | SR: AndroidControl average +0.4 points; OmniAct average +0.3 points | Helped |
| Fine-tune OS-Atlas-4B on OmniBench graph-structured trajectories | OS-Atlas-4B · AndroidControl and OmniAct | SR: AndroidControl average +0.46 points; OmniAct average +0.73 points | Helped |
| Train on OmniBench graph-structured trajectories rather than standard model training | Omni-OS-Atlas and Omni-Aguvis · Instruction-order sensitivity | Avg. Sensitivity: Omni-OS-Atlas 9.07 to 3.49; Omni-Aguvis 12.9 to 2.67 | Helped |
| Increase task complexity from Easy to Medium to Hard | Virtual agents · OmniBench tasks across five complexity dimensions | Performance: average decrease of 6.19 points | Hurt |
| Use graph-structured tasks rather than chain-structured tasks | GPT-4o with A11Y · OmniBench graph-structured tasks | Accuracy: 20.5% on graph-structured tasks; human performance 80.1% | Hurt |
| Compare Omni-UGround-V1-7B with its UGround-7B-V1 baseline | UGround-7B-V1 · OmniBench | Coverage Rate (CR): 25.0 to 34.4 | Helped |
Surprising findings
- Adding task intents to prompts improves GPT-4o planning from 25.4 to 34.3 overall.
- Agents fine-tuned on chain-structured data tend to misread graph tasks as linear, a major failure mode.
Limitations
- Human acceptance measured on sampled task graphs by three annotators, and the evaluation depends on GPT-4o/Claude-generated synthetic data.
- Evaluation is limited to a Windows desktop virtual environment with 49 applications in the main experiments.
Experimental setup
Datasets: OmniBench synthesized graph-structured tasks AndroidControl OmniAct
Benchmarks: OmniBench AndroidControl OmniAct
Metrics: Coverage Rate Logical Consistency success rate grounding accuracy action type accuracy sensitivity (std)
Baselines: Qwen2-VL-7B InternVL2-8B InternVL2.5-8B Qwen-VL-Max Gemini-2.0-Flash Claude-3.5-Sonnet GPT-4o Aguvis-7B OS-Atlas-Pro-4B ShowUI-2B OS-Atlas-Base-4B UGround-V1-7B
Builds on: OSWorld CRAB OS-Atlas UGround Aguvis
Closest papers by meaning
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.86)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.86)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.86)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.85)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.85)
- VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.