Research map / Multi-agent systems and games / LLM agents
Aegis: Automated Error Generation and Identification for Multi-Agent Systems
Fanqi Kong, Ruijie Zhang, Huaxiao Yin, Guibin Zhang, Xiaofei Zhang, Ziang Chen, Zhaowei Zhang, Xiaoyuan Zhang, Song-Chun Zhu, Xue Feng
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Aegis, a framework for automatically generating and identifying errors in LLM-based multi-agent systems to create large-scale error attribution datasets instead of costly manual annotation.
Problem
Multi-agent LLM systems are fragile and hard to debug, and large diverse datasets for error attribution are scarce because existing resources depend on costly, unscalable manual annotation.
Approach
Automates both the generation of errors in multi-agent system trajectories and the identification (attribution) of those errors, replacing manual annotation with scalable automated error injection and labeling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with SFT on Aegis data | Qwen2.5-14B-Instruct · Aegis-Bench + Who&When avg F1 | avg F1: 13.99 -> 26.51 | Improved |
| fine-tune with GRPO and hierarchical reward on Aegis data | Qwen2.5-14B-Instruct · avg F1 | avg F1: 13.99 -> 18.41 | Improved |
| SFT on Aegis data | Qwen2.5-7B-Instruct · Who&When Pair | micro-F1: 2.31 -> 1.26 | Hurt |
| SFT on Aegis data | Qwen3-8B-Non-Thinking · avg F1 | avg F1: 10.12 -> 21.41 | Improved |
| GRPO on Aegis data | Qwen3-8B-Thinking · avg F1 | avg F1: 13.06 -> 17.58 | Improved |
| remove hierarchical consistency loss (ablation) | DCL · avg F1 | avg F1: 12.61 -> 9.52 | Hurt |
| remove text-prototype intent guidance (ablation) | DCL · avg F1 | avg F1: 12.61 -> 10.16 | Hurt |
| use only-bilinear head instead of compositional head (ablation) | DCL · avg F1 | avg F1: 12.61 -> 10.01 | Hurt |
| add chain-of-thought prompting to base open-source models | Qwen base models · Aegis error attribution | avg F1 | Hurt |
| add chain-of-thought after GRPO training | GRPO-trained Qwen models · Aegis error attribution | avg F1 | Improved |
| train SFT beyond two epochs (ablation) | Aegis-SFT · Who&When | avg F1 | Hurt |
| SFT on Aegis for small MAS (Magentic-One, SmolAgents) | Aegis-SFT · Aegis-Bench small MAS | avg score | Hurt |
Closest papers by meaning
- CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems (ICML 2026 · similarity 0.92)
- Conformal Agent Error Attribution (NeurIPS 2026 · similarity 0.90)
- Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution (NeurIPS 2026 · similarity 0.90)
- AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? (ICLR 2026 · similarity 0.89)
- DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.89)
- AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
