Research map / Multi-agent systems and games / LLM agents

Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies security risks of LLM-based end-to-end software development multi-agent systems via an implicit malicious behavior injection attack (IMBIA) under malicious-user and benign-user-with-malicious-agent scenarios, and proposes an adversarial-prompt defense (Adv-IMBIA).

Problem

End-to-end LLM multi-agent software development systems can be exploited to generate software with hidden malicious functionality, either by malicious users giving requirements or by compromised agents, and these risks are largely unexplored.

Approach

Append a tripartite malicious task (summary, description, code instructions) to benign software requirements (MU-BA) or inject it into agent profiles (BU-MA), then defend with adversarial prompts placed in agent profiles or the user interface, and analyze which development phases and agents are most critical.

Key results

• IMBIA achieved ASRs of 93%, 45%, and 71% on ChatDev, MetaGPT, and AgentVerse in the MU-BA scenario.
• In the BU-MA scenario, IMBIA reached ASRs of 71%, 84%, and 45% on ChatDev, MetaGPT, and AgentVerse.
• Adv-IMBIA reduced MU-BA ASR by 73%, 40%, and 49% (relative reductions reported) for ChatDev, MetaGPT, and AgentVerse; GPT-4o judge agreed with human evaluators at 86.34% on average.

Surprising findings

• Attacking all agents simultaneously did not yield the highest attack success, suggesting conflicts between benign and malicious tasks cause agent hesitation.
• More advanced base models were generally more vulnerable to IMBIA; GPT-4o-mini had the highest ASR (0.929) in the base model comparison.

Limitations

• Evaluation mostly relies on GPT-4o-mini and a GPT-4o judge with only partial human validation (407 samples on ChatDev MU-BA).
• Malicious behaviors are limited to 12 predefined malware-style behaviors from five families.
• Defense effectiveness is notably weaker in the BU-MA scenario.

Experimental setup

Datasets: SRDD

Benchmarks: SRDD-derived test set of 480 cases

Metrics: attack success rate reject rate benign utility utility under attack consistency

Baselines: ChatDev MetaGPT AgentVerse

Builds on: ChatDev MetaGPT AgentVerse SRDD

Closest papers by meaning

• ORBIT: A Framework for Multi-Agent Security Evaluations (NeurIPS 2026 · similarity 0.90)
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.88)
• When Embedding-Based Defenses Fail: Rethinking Safety in LLM-Based Multi-Agent Systems (ICML 2026 · similarity 0.88)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.88)
• Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools (NeurIPS 2025 · similarity 0.87)
• OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.87)