Research map / Multi-agent systems and games / LLM agents

Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Agora, a domain-aware three-agent LLM framework (Orchestrator, Strategy, TestGen) following hypothesis-driven testing to find protocol-level logic bugs in four consensus implementations, reporting 15 zero-day logic bugs versus implementation-only findings from ReAct baselines.

Problem

Detecting deep, state-dependent protocol-level logic bugs in consensus protocol implementations (Raft, EPaxos, HotStuff, BullShark), which LLM-based code analysis tools struggle with.

Approach

A multi-agent workflow that decouples workflow control, domain-constrained attack scenario synthesis (CFT vs BFT constraints and bug-pattern knowledge), and reflection-based unit test generation, operationalizing hypothesis-driven testing with LLMs.

Key results

• Agora with Gemini 3.0 Pro Preview finds 11 logic bugs; across all four LLMs it finds 15 unique zero-day logic bugs, while all ReAct baselines find zero logic bugs.
• Baselines find 22 implementation-level bugs (mostly GPT-5.2), while Agora reports none by design.
• Of 46 bug reports, 34 were real logic bugs, giving a reported false positive rate of 26.1%.

Surprising findings

• Gemini and Claude ReAct baselines found no bugs at all, and the baseline bugs found were implementation-level rather than logic bugs.
• Removing the scenario-generator yields 29 implementation-level bugs but zero logic bugs.

Limitations

• Relies on manually curated domain knowledge and bug patterns, requiring human effort.
• Small number of bugs (15) and only four repositories limit generality.
• Evaluation counts are small and baselines are ad hoc ReAct adaptations, so comparisons may be sensitive to setup.

Experimental setup

Datasets: 816 GitHub issues from four consensus repositories (34 used for bug patterns)

Benchmarks: etcd Raft efficientEPaxos relabHotStuff sui BullShark

Metrics: number of protocol-level logic bugs found number of implementation bugs found false positive rate token consumption

Baselines: ReAct with GPT-5.2 ReAct with Gemini 3.0 Pro Preview ReAct with Claude Sonnet 4.5 ReAct with Qwen3 Coder 480B A35B

Builds on: hypothesis-driven testing ReAct MetaGPT Model Context Protocol

Closest papers by meaning

• Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerance (AAAI 2026 · similarity 0.86)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.85)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.85)
• Verify0: Can AI Agents Build Formally Verified Software Repositories? (NeurIPS 2026 · similarity 0.85)
• ProtocolBench: Which LLM MultiAgent Protocol to Choose? (ICML 2026 · similarity 0.85)
• ContractBench: Can LLM Agents Preserve Observation Contracts? (NeurIPS 2026 · similarity 0.84)