Research map / Multi-agent systems and games / LLM agents
SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories
Mingxuan (Aldous) Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph Gonzalez, Alex Dimakis
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SlackBench, a benchmark for evaluating AI agents on collaborative project work grounded in real code repositories and multi-party conversations.
Problem
Existing agent benchmarks rely on synthetic data or isolated tasks and miss the multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions.
Approach
A benchmark built from real code repositories paired with collaborative conversation context, so agents must integrate scattered information across chat, code, and evolving decisions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| give agents direct CLI access to exported workspace files instead of MCP tools (ablation) | coding agents · SlackBench overall | accuracy (%): average +7.85%; Codex GPT-5.5 45.52 -> 59.90 | Improved |
| use direct CLI access instead of MCP (ablation) | Claude Code with Opus 5.5 · SlackBench | cost: 54% lower cost | Improved |
| use direct CLI access instead of MCP (ablation) | other agents · SlackBench | token consumption: 21.00% fewer tokens on average | Improved |
| raise reasoning effort from low to high (ablation) | Claude Code with Claude Opus 4.7 · SlackBench overall | accuracy (%): 48.80 -> 67.62 | Improved |
| raise reasoning effort from medium to high (ablation) | Codex CLI with GPT-5.4-mini · SlackBench overall | accuracy (%): 27.04 -> 27.47 | No clear effect |
| raise reasoning effort from medium to high (ablation) | Claude Code with Claude Sonnet 4.6 · SlackBench Hallucination | accuracy (%): 40.97 -> 33.33 | Hurt |
| use official Slack MCP instead of the SlackBench mock MCP (ablation) | Claude Code with Claude Sonnet 4.6 · SlackBench AS filtered | accuracy (%): 46.67 -> 57.78 | Improved |
| run Muse Wonder in Claude Code instead of Muse Code (ablation) | Muse Wonder · SlackBench overall | accuracy and cost: 73.74 -> 70.72; cost $143 -> over $500 | Hurt |
| use GPT models for false-premise questions | GPT 6 Astra and GPT 6.1 Sol · SlackBench Hallucination | answer-first failure rate: 25% and 21% vs 8% for Opus 5.5 | Hurt |
Experimental setup
Benchmarks: SlackBench
Closest papers by meaning
- SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering (ICML 2025 · similarity 0.89)
- Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.89)
- AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators (NeurIPS 2026 · similarity 0.88)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.88)
- AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering (NeurIPS 2026 · similarity 0.88)
- ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
