Research map / Multi-agent systems and games / LLM agents

ProtocolBench: Which LLM MultiAgent Protocol to Choose?

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ProtocolBench, a four-axis benchmark comparing LLM multi-agent communication protocols (A2A, ACP, ANP, Agora) across four scenarios, and ProtocolRouter, a constraint-aware router that selects per-scenario or per-module protocols.

Problem

Choice of LLM multi-agent communication protocol (A2A, ACP, ANP, Agora) is intuition-driven, with no standardized comparison of task success, latency, overhead, robustness and security.

Approach

A protocol-agnostic benchmark with thin native adapters and shared scenario harnesses isolates protocol effects, plus a constraint-first router that filters protocols by hard requirements and breaks ties with performance priors.

Key results

• In Streaming Queue, ACP has the lowest mean latency (9.66 s) and total duration varies by up to 36.5% across protocols (40.28 vs 54.97 min).
• ProtocolRouter reduces Fail-Storm recovery time from 8.00 s (A2A) to 6.55 s (18.1%) and raises GAIA success from 9.29 to 9.90.
• On ProtocolRouterBench, spec+perf routing reaches 63.3% scenario accuracy and 81.7% module accuracy versus 53.5% and 71.2% for spec-only.

Surprising findings

• A2A has the highest GAIA task utility but ACP has the lowest Streaming Queue latency; no single protocol dominates across scenarios.
• ANP and Agora cover all five security dimensions in Safety Tech, while A2A and ACP fail TLS and tunnel sniffing probes.

Limitations

• All main experiments use a single open-source model (Qwen2.5-VL-72B-Instruct); cross-model conclusions are not claimed.
• ProtocolRouter is an offline planner and does not handle highly dynamic or adversarial environments or cold-start protocols.
• Scale-up experiments are limited to moderate scale (up to 32 agents) and only 10 Safety cases and 1000 MS MARCO entries.

Experimental setup

Datasets: GAIA ChatDoctor-HealthCareMagic-100k MS MARCO 2WikiMultihopQA

Benchmarks: GAIA Fail-Storm Recovery Streaming Queue Safety Tech ProtocolRouterBench

Metrics: success average quality average (1-5) mean end-to-end latency std dev recovery time answer discovery rate probe block rate scenario accuracy module accuracy macro-F1

Baselines: A2A ACP ANP Agora single-protocol deployments

Builds on: A2A ACP ANP Agora MCP

Closest papers by meaning

• Augmented Runtime Collaboration for Self-Organizing Multi-Agent Systems: A Hybrid Bi-Criteria Routing Approach (AAAI 2026 · similarity 0.85)
• ContractBench: Can LLM Agents Preserve Observation Contracts? (NeurIPS 2026 · similarity 0.85)
• Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents (ICML 2026 · similarity 0.85)
• AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators (NeurIPS 2026 · similarity 0.84)
• Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerance (AAAI 2026 · similarity 0.84)
• A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliability (NeurIPS 2026 · similarity 0.84)