Research map / Multi-agent systems and games / LLM agents

Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Workspace-Bench 1.0, a benchmark of 388 file-dependency-driven office tasks across five large persona workspaces (about 20k files), and evaluates 15 agent harness and LLM configurations with rubric-based Agent-as-a-Judge scoring.

Problem

Existing agent benchmarks use pre-specified or synthetic small file sets with limited real-world inter-file dependencies, so they do not test workspace-level reasoning over explicit and implicit file dependencies such as lineage and semantic relations.

Approach

Builds large, persona-driven, deeply nested file workspaces with heterogeneous file types, human-curated tasks with file dependency graphs and fine-grained rubrics, and an evaluation harness (sandbox pool, workspace rollback, Agent-as-a-Judge with dependency-graph recognition).

Key results

• Average rubric pass rate across 27 harness-LLM combinations is 43.3%; best combination DeepAgent + GLM-5.1 reaches about 61% total.
• Performance degrades from Easy (57.6%) to Hard (40.5%) tasks across configurations.
• Human-agent collaboration reaches 80.7%, well above fully autonomous execution.

Surprising findings

• Open-source DeepAgent + MiniMax-M2.7 shows severe cost explosion, using up to 58.1 turns and 0.61M tokens per task while averaging only 45% pass rate.
• Harnesses have limited impact on strong models but substantially boost weaker foundation models.

Limitations

• Human-agent collaboration and some results are reported in ways that may be inconsistent across tables (e.g., 15 vs 27 configurations).
• Workspaces are synthesized by LLMs and crawled resources, which may limit realism.

Experimental setup

Datasets: Workspace-Bench 1.0 workspaces (20,476 files across five personas)

Benchmarks: Workspace-Bench 1.0 Workspace-Bench 1.0-Lite

Metrics: rubric pass rate task completion rate pass@k dependency graph recognition rate (F1) token consumption interaction turns

Baselines: DeepAgent Hermes OpenClaw Codex

Builds on: OSWorld TheAgentCompany OfficeBench

Closest papers by meaning

• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.89)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.87)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.87)
• PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.87)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.87)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.87)