Research map / Multi-agent systems and games / LLM agents
DDBench: A Benchmark for Agentic Debugging on Distributed Systems
Yibo Yan, Huijuan Wang, Junzhou He, Yizhuo Liang, Shaoyu Wang, Huanchen Sun, Seo Jin Park
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DDBench, a benchmark for evaluating LLM coding agents on debugging bugs in distributed systems, where root causes span processes, nodes, and protocol interactions.
Problem
LLM coding agents perform well on single-process SWE tasks, but debugging distributed systems remains under-explored because bugs span processes and nodes, root causes are rarely recoverable from source alone, and non-deterministic interleavings make brute-force exploration intractable.
Approach
A benchmark targeting agentic debugging of distributed-system bugs, distinct from single-process SWE-bench-style tasks; the abstract indicates it addresses cross-process, cross-node, and protocol-level failures. Specific method details are not stated in the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add debug-context bundle (logs, traces, notes) (ablation) | 10 LLMs with mini-swe-agent · DDBench tier-1 | aggregate pass rate: 31.0% -> 50.0% (+19.0 pp) | Improved |
| add debug-context bundle (ablation) | GLM 4.7 · DDBench tier-1 | pass rate: 9.7% -> 48.4% | Improved |
| add debug-context bundle (ablation) | Claude Opus 4.6 · DDBench tier-1 | pass rate: 51.6 -> 64.5 (p=0.13) | No clear effect |
| add debug-context bundle (ablation) | Claude Opus 4.6 · DDBench tier-1 | tokens per case: 3.11M -> 1.21M (-61%) | Improved |
| add debug-context bundle (ablation) | 10 LLMs · DDBench tier-1 | mean steps: 57.6 -> 43.1 | Improved |
| add debug-context bundle (ablation) | 10 LLMs · DDBench tier-2 | aggregate pass rate: 48.0% -> 62.7% | Improved |
| add debug-context bundle (ablation) | 10 LLMs · DDBench tier-3 | aggregate pass rate: 67.9% -> 77.1%; two models -7.1 pp | Mixed |
| add debug-context bundle (ablation) | 10 LLMs · DDBench tier-1 concurrency hazards | pass rate: 30.0% -> 75.0% | Improved |
| add debug-context bundle (ablation) | 10 LLMs · DDBench tier-1 protocol and recovery | pass rate: 27.7% -> 39.2% | Improved |
| provide debug-context far from the fault (large abstraction distance) (ablation) | Claude Opus 4.6 · Kvrocks-3356 case | debugging path | Hurt |
| move from SWE-bench Verified to DDBench tier-1 for model comparison | six frontier models · pass rate spread | spread across models: 7 pp -> 42 pp | Improved |
Experimental setup
Benchmarks: DDBench
Builds on: SWE-bench Verified
Closest papers by meaning
- DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.90)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.88)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.88)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.88)
- AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators (NeurIPS 2026 · similarity 0.88)
- Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
