Research map / Multi-agent systems and games / LLM agents
Self-Evolving Multi-Agent Collaboration Networks for Software Development
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvoMAC, a test-time self-evolving multi-agent coding network updated via textual backpropagation driven by compiler/unit-test feedback, and rSDE-Bench, a requirement-oriented software benchmark with automatic test-based evaluation.
Problem
Multi-agent LLM coding systems rely on human-designed static workflows that cannot adapt to diverse real-world software tasks, and existing benchmarks focus on functions or bug fixing rather than requirement-level software development with automatic correctness checking.
Approach
Treat a MAC network's agents (prompts) and workflow topology as trainable parameters updated at test time: unit tests generated by a separate testing MAC network act as a target proxy, compiler execution logs serve as textual loss, and a gradient agent plus an updating agent perform textual backpropagation to revise the coding team.
Key results
- EvoMAC (GPT-4o-Mini) reaches 89.38% on rSDE-Bench Website Basic, a +26.48 point gain over the best single-agent baseline (GPT-4o-Mini, 62.90%).
- EvoMAC achieves 94.51% pass@1 on HumanEval, versus 90.85% for the best multi-agent baselines (MapCoder, AgentVerse).
- Automatic accuracy metric in rSDE-Bench correlates 0.9922 with human evaluation, compared with 0.2583 for consistency and 0.3041 for quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EvoMAC instead of ChatDev | GPT-4o-Mini coding systems · rSDE-Bench Website Basic | accuracy (%): 62.67±0.28 -> 89.38±1.01; +26.48 | Helped |
| Use EvoMAC instead of GPT-4o-Mini single-agent | GPT-4o-Mini coding systems · rSDE-Bench Website Advanced | accuracy (%): 44.40±4.21 -> 65.05±1.56; +20.65 | Helped |
| Use EvoMAC instead of ChatDev | GPT-4o-Mini coding systems · rSDE-Bench Game Basic | accuracy (%): 53.63±5.70 -> 77.54±2.04; +34.78 | Helped |
| Use EvoMAC instead of ChatDev | GPT-4o-Mini coding systems · rSDE-Bench Game Advanced | accuracy (%): 32.26±4.55 -> 51.60±4.54; +21.50 | Helped |
| Use EvoMAC instead of MapCoder | GPT-4o-Mini coding systems · HumanEval | Pass@1 (%): 90.85 -> 94.51; +6.10 | Helped |
| Add objective environment feedback in place of LLM critique (ablation) | EvoMAC · rSDE-Bench Website Basic | accuracy (%): 78.08 -> 90.75 | Helped |
| Add objective environment feedback in place of LLM critique (ablation) | EvoMAC · rSDE-Bench Website Advanced | accuracy (%): 52.23 -> 67.20 | Helped |
| Add objective environment feedback in place of LLM critique (ablation) | EvoMAC · rSDE-Bench Game Basic | accuracy (%): 55.80 -> 77.54 | Helped |
| Add objective environment feedback in place of LLM critique (ablation) | EvoMAC · rSDE-Bench Game Advanced | accuracy (%): 33.32 -> 51.60 | Helped |
| Use multi-agent coding instead of single-agent coding (ablation) | EvoMAC with multi-agent testing, evolution and environment feedback · rSDE-Bench Website Basic | accuracy (%): 83.56 -> 90.75 | Helped |
| Use multi-agent coding instead of single-agent coding (ablation) | EvoMAC with multi-agent testing, evolution and environment feedback · rSDE-Bench Website Advanced | accuracy (%): 61.94 -> 67.20 | Helped |
| Use multi-agent testing instead of single-agent testing (ablation) | EvoMAC with multi-agent coding, evolution and environment feedback · rSDE-Bench Website Basic | accuracy (%): 83.90 -> 90.75 | Helped |
Surprising findings
- Replacing compiler/environment feedback with an LLM critic lowers performance by 12-22 points, suggesting LLM self-critique can bias evolution.
- Consistency and quality metrics from prior SRDD work correlate weakly with human judgments of requirement fulfillment.
Limitations
- rSDE-Bench is small: 53 tasks (45 websites, 8 games) with a limited set of game types.
- Relies on a strong proprietary LLM (GPT-4o-Mini, Claude-3.5-Sonnet) and multiple evolving iterations, with associated API cost.
- Test-case generation for websites uses gpt-4o-mini with manual correction, and game tests are manually written.
Experimental setup
Datasets: HumanEval rSDE-Bench
Benchmarks: rSDE-Bench HumanEval
Metrics: accuracy (test pass rate) pass@1 consistency quality correlation with human evaluation
Baselines: GPT-4o-Mini Claude-3.5-Sonnet Gemini-1.5-Flash MetaGPT AutoGen MapCoder AgentVerse ChatDev
Builds on: MetaGPT ChatDev MapCoder SRDD
Closest papers by meaning
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.87)
- EvoMAS: Evolutionary Generation of Multi-Agent Systems (ICML 2026 · similarity 0.86)
- CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments (ICML 2026 · similarity 0.86)
- daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.86)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.86)
- Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.