Research map / Multi-agent systems and games / LLM agents

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AgentCollabBench, a 900-task diagnostic benchmark measuring four behavioral failure modes (instruction decay, tracer durability, consensus pollution, cross-task leakage) in multi-agent LLM systems across communication topologies, evaluating four LLMs.

Problem

Outcome-based evaluations of multi-agent LLM systems miss process-level failures such as dropped constraints, lost information across hops, false-belief spread, and cross-task leakage.

Approach

A controlled-injection benchmark with four deterministic/LLM-judged metrics (IDR, RTD, CPR, CLC) stratified across domains, topologies, and complexity tiers, plus a finding that converging-DAG nodes drop constraints from minority parent branches.

Key results

• Qwen-3.5-35B-A3B leads on tracer durability (RTD 94.0%) and instruction decay (IDR 0.9%), while GPT 4.1 mini has the lowest cross-task leakage (CLC 2.6%) and consensus pollution (CPR 17.7%).
• Topology explains 7-40% of variance (eta^2 = 0.07-0.40) in RTD; converging-DAG hops drop tracers at 1.6-5.0x the rate of linear-chain hops.
• Llama 3.1 8B Instruct is worst on three of four metrics (IDR 10.1%, CPR 40.3%, RTD 62.6%).

Surprising findings

• Converging-DAG topology causes substantially higher constraint loss than linear chains across all four models, with effect size comparable to backbone choice.
• Structural complexity tiers are largely orthogonal to behavioral failure rates; CLC even decreases with complexity for GPT 4.1 mini.

Limitations

• Coverage limited to three domains and five fixed topologies.
• Potential evaluator bias from LLM-judged metrics (IDR, CPR) despite judge-backbone separation.
• No analysis of correlation between metrics and task success.

Experimental setup

Datasets: AgentCollabBench

Benchmarks: AgentCollabBench

Metrics: Instruction Decay Rate Radioactive Tracer Durability Consensus Pollution Rate Cross-task Leakage Containment Cohen's kappa Spearman correlation Kruskal-Wallis eta-squared

Baselines: Chatbot Arena leaderboard ranks

Builds on: MultiAgentBench GEMMAS MAST

Closest papers by meaning

• Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.89)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.89)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents (ICML 2025 · similarity 0.88)
• The Best-Laid SCHEMEs: Coordinated Sabotage and Monitoring in Multi-Agent Systems (NeurIPS 2026 · similarity 0.88)
• AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems (NeurIPS 2026 · similarity 0.88)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.88)