Research map / Multi-agent systems and games / LLM agents
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TheAgentCompany, a self-hosted benchmark of 175 checkpoint-scored software-company tasks that tests LLM agents on browsing, coding, terminal use, and colleague communication; the best model completes 30.3% of tasks.
Problem
Existing agent benchmarks rarely cover realistic multi-tool professional work with communication, long horizons, and self-hosted reproducibility, so it is unclear how well LLM agents can automate consequential workplace tasks.
Approach
A simulated software company intranet (GitLab, ownCloud, Plane, RocketChat) with LLM-simulated colleagues, 175 tasks across SWE/PM/DS/Admin/HR/Finance, checkpoint-based partial-credit scoring, and mixed deterministic plus LLM-based evaluators.
Key results
- Gemini-2.5-Pro with OpenHands completes 30.3% of 175 tasks fully and scores 39.3% on partial credit.
- Claude-3.7-Sonnet reaches 26.3% success and Claude-3.5-Sonnet 24.0%, while GPT-4o reaches 8.6%.
- Best open-weights model Llama-3.1-405b reaches 7.4% success, with Llama-3.3-70b close at 6.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini-2.5-Pro instead of Claude-3.7-Sonnet with OpenHands | OpenHands · TheAgentCompany (175 tasks) | Success: 26.3% -> 30.3% | Helped |
| Use Gemini-2.5-Pro instead of Claude-3.7-Sonnet with OpenHands | OpenHands · TheAgentCompany (175 tasks) | Score: 36.4% -> 39.3% | Helped |
| Use Claude-3.7-Sonnet instead of Claude-3.5-Sonnet with OpenHands | OpenHands · TheAgentCompany (175 tasks) | Success: 24.0% -> 26.3% | Helped |
| Use Gemini-2.0-Flash instead of GPT-4o with OpenHands | OpenHands · TheAgentCompany (175 tasks) | Score: 16.7% -> 19.0% | Helped |
| Use OpenHands CodeAct instead of OWL RolePlay with GPT-4o | GPT-4o agents · TheAgentCompany (175 tasks) | Success: 4.0% -> 8.6% | Helped |
| Use Llama-3.1-405b instead of GPT-4o with OpenHands | OpenHands · TheAgentCompany (175 tasks) | Success: 8.6% -> 7.4% | Hurt |
| Use Llama-3.3-70b instead of Llama-3.1-70b with OpenHands | OpenHands · TheAgentCompany (175 tasks) | Success: 1.7% -> 6.9% | Helped |
| Use Gemini-2.0-Flash instead of Gemini-2.5-Pro with OpenHands | OpenHands · TheAgentCompany (175 tasks) | Costs: $4.2 -> $0.6 | Helped |
Surprising findings
- Admin, finance, and data science tasks score lower than software engineering tasks despite seeming easier for humans.
- Gemini 2.0 Flash is cheap (under $1 per task) but often loops or explores aimlessly, needing around 40 steps.
Limitations
- Tasks are mostly straightforward and automatically verifiable, excluding creative or architectural work.
- Only two agent scaffolds (OpenHands CodeAct and OWL RolePlay) were used as baselines.
- No human professional baseline was measured for comparison.
Experimental setup
Datasets: O*NET 29.1
Benchmarks: TheAgentCompany
Metrics: full completion score partial completion score steps cost per instance
Baselines: OWL RolePlay Claude-3.7-Sonnet Claude-3.5-Sonnet Gemini-2.0-Flash GPT-4o Gemini-1.5-Pro Amazon-Nova-Pro-v1 Llama-3.1-405b Llama-3.3-70b Qwen-2.5-72b
Builds on: WebArena OpenHands Sotopia O*NET
Closest papers by meaning
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.89)
- WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? (ICML 2024 · similarity 0.89)
- ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems (CVPR 2025 · similarity 0.88)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.88)
- SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? (NeurIPS 2026 · similarity 0.87)
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.