Research map / Multi-agent systems and games / LLM agents
DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DoVer, an intervention-driven debugging framework that segments failed multi-agent logs into trials, generates failure hypotheses, and validates them by editing messages/plans and re-running from the intervention point.
Problem
Log-only failure attribution in LLM multi-agent systems yields untested hypotheses, and single-step/single-agent attribution is ill-posed because multiple interventions can repair the same failure and ground-truth labels are uncertain.
Approach
Treat failure attribution as a hypothesis to validate: segment logs into planning-execution trials, propose a faulty step per trial, apply a targeted intervention (edit orchestrator message or plan) in place, re-execute, and judge via task success, milestone progress, and LLM-judged intervention fulfillment; evaluate outcome recovery rather than attribution accuracy.
Key results
- Trial success rate of 17.6% on WW-AB and WW-GAIA and 27.5% on GAIA-Level-1 under Magentic-One, i.e. roughly 18-28% of failed trials flipped to success.
- On GSMPlus with AG2 MathChat, DoVer achieves a 49.0% trial success rate.
- Progress made reaches +15.7% on GAIA-Level-1; 30-60% of hypotheses validated or refuted, with Validated at 34.9% on GAIA-Level-1, while Self-Refine and CRITIC recover 0% of WW-GAIA failures.
Surprising findings
- Self-Refine and CRITIC-style self-improvement baselines flip none of the 26 failed WW-GAIA cases.
- Step-level attribution on Who&When is below 25% even with step indices, guidance, and GPT-5, with about half of GAIA GT labels judged uncertain.
Limitations
- Interventions restricted to orchestrator messages; sub-agent capabilities and tools are not modified.
- Evaluation covers only two frameworks, web-search and math tasks, and GPT-4o/Qwen3 models.
- Milestone progress and intervention-fulfillment labels rely on LLM-as-a-judge, which may be biased.
Experimental setup
Datasets: Who&When (WW-AB, WW-GAIA) GAIA AssistantBench GSMPlus
Benchmarks: WW-AB WW-GAIA GAIA-Level-1 GSMPlus
Metrics: trial success rate progress made validated/refuted/inconclusive hypothesis rate step-level attribution accuracy
Baselines: Self-Refine CRITIC All-at-Once attribution (Who&When)
Builds on: AGDebugger Who&When failure attribution Magentic-One AutoGen MAST
Closest papers by meaning
- AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? (ICLR 2026 · similarity 0.91)
- Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · similarity 0.90)
- Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (ICML 2025 · similarity 0.90)
- Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution (NeurIPS 2026 · similarity 0.90)
- CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems (ICML 2026 · similarity 0.89)
- MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.