AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

Mingxuan (Aldous) Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph Gonzalez, Alex Dimakis

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SlackBench, a benchmark for evaluating AI agents on collaborative project work grounded in real code repositories and multi-party conversations.

Problem

Existing agent benchmarks rely on synthetic data or isolated tasks and miss the multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions.

Approach

A benchmark built from real code repositories paired with collaborative conversation context, so agents must integrate scattered information across chat, code, and evolving decisions.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
give agents direct CLI access to exported workspace files instead of MCP tools (ablation)coding agents · SlackBench overallaccuracy (%): average +7.85%; Codex GPT-5.5 45.52 -> 59.90Improved
use direct CLI access instead of MCP (ablation)Claude Code with Opus 5.5 · SlackBenchcost: 54% lower costImproved
use direct CLI access instead of MCP (ablation)other agents · SlackBenchtoken consumption: 21.00% fewer tokens on averageImproved
raise reasoning effort from low to high (ablation)Claude Code with Claude Opus 4.7 · SlackBench overallaccuracy (%): 48.80 -> 67.62Improved
raise reasoning effort from medium to high (ablation)Codex CLI with GPT-5.4-mini · SlackBench overallaccuracy (%): 27.04 -> 27.47No clear effect
raise reasoning effort from medium to high (ablation)Claude Code with Claude Sonnet 4.6 · SlackBench Hallucinationaccuracy (%): 40.97 -> 33.33Hurt
use official Slack MCP instead of the SlackBench mock MCP (ablation)Claude Code with Claude Sonnet 4.6 · SlackBench AS filteredaccuracy (%): 46.67 -> 57.78Improved
run Muse Wonder in Claude Code instead of Muse Code (ablation)Muse Wonder · SlackBench overallaccuracy and cost: 73.74 -> 70.72; cost $143 -> over $500Hurt
use GPT models for false-premise questionsGPT 6 Astra and GPT 6.1 Sol · SlackBench Hallucinationanswer-first failure rate: 25% and 21% vs 8% for Opus 5.5Hurt

Experimental setup

Benchmarks: SlackBench

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy