Research map / Multi-agent systems and games / LLM agents
UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UniClawBench, a capability-driven benchmark of 400 bilingual real-world tasks evaluating proactive agents in live Docker containers with a hidden-supervisor, user-simulator closed-loop multi-turn evaluation, and compares 10 models and 3 agent frameworks.
Problem
Existing agent benchmarks rely on sandboxed or cached environments, single-turn evaluation, and scenario-based taxonomies that conflate model capabilities, making failure root causes hard to diagnose for proactive agents.
Approach
Organize tasks by five foundational capabilities (skill usage, exploration, long-context reasoning, multimodal understanding, cross-platform coordination) and evaluate in live Docker containers with a three-role loop where a hidden supervisor checks checkpoint rubrics and only a coarse status signal reaches a user simulator, which provides sanitized feedback to the executor.
Key results
- Claude Opus-4.8 achieves the highest overall pass rate at 47.5% under OpenClaw, and GPT-5.4 the highest overall average score at 0.774; all models remain below 50% pass rate.
- Under GPT-5.4, OpenClaw achieves 40.7% PR versus 33.8% for EDICT and 29.0% for Nanobot, showing framework choice strongly affects results.
- Automatic pass/fail agrees with human majority vote on 92.0% of 50 sampled trajectories; checkpoint AS correlates with human scores at Pearson r=0.71.
Surprising findings
- Framework choice exerts a stronger influence than model choice, per the authors.
- Most models achieve high checkpoint scores yet fail to complete tasks (a 'halfway failure' phenomenon).
Limitations
- Relatively small set of 400 manually curated tasks.
- Possible instability from live-environment execution.
- Potential bias from LLM-based evaluation (supervisor and user simulator).
Experimental setup
Benchmarks: UniClawBench
Metrics: Pass Rate Average Score (checkpoint-based) token usage
Baselines: ClawBench SkillsBench ClawMark EDICT Nanobot
Builds on: ReAct OpenClaw Nanobot WebArena OSWorld
Closest papers by meaning
- ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response (ICML 2026 · similarity 0.91)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.90)
- Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.90)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.90)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.90)
- MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.