Research map / Multi-agent systems and games / LLM agents

Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Co-RedTeam, a security-aware multi-agent LLM framework that runs vulnerability discovery and execution-grounded iterative exploitation with layered long-term memory, evaluated on CyBench, BountyBench and CyberGym.

Problem

Automating software vulnerability discovery and exploitation with LLMs is hard: single-shot or generic coding agents lack execution grounding, security-domain knowledge, and the ability to reuse past experience.

Approach

An orchestrator coordinates Analysis and Critique agents for evidence-grounded vulnerability discovery, then Planner, Validation, Execution and Evaluation agents run a closed plan-execute-evaluate loop in Docker, with a three-layer memory (vulnerability patterns, strategies, technical actions) that accumulates across tasks.

Key results

• With Gemini-3-pro, Co-RedTeam reaches 63.7% on CyBench, 65.0% exploit and 20.0% detect on BountyBench, and 37.3% on CyberGym, above the strongest baselines.
• Removing execution drops CyBench success from 59.1% to 17.5% with Gemini-2.5-pro.
• Memory evolution on CyberGym shows rising moving-average success while static or no-memory settings plateau.

Surprising findings

• Static analysis baselines VulTrail and RepoAudit underperform generic coding agents on exploitation despite stronger detection-oriented design.
• Gemini-3-pro converges in about 13 iterations, while Gemini-2.5-pro keeps improving until about 17.

Limitations

• Security knowledge base is a curated static CWE/OWASP summary; the authors note a search engine could replace it.
• Detection precision and recall remain low in absolute terms (precision 14.3%, recall 12.5% with Gemini-2.5-pro).
• Evaluation covers only Gemini backbones in the main results; other model families are only in the appendix.

Experimental setup

Datasets: CyBench BountyBench CyberGym CWE/OWASP-derived vulnerability knowledge base

Benchmarks: CyBench BountyBench CyberGym

Metrics: success rate precision recall latency (s)

Baselines: Vanilla OpenHands C-Agent VulTrail RepoAudit

Builds on: Co-Scientist OpenHands

Closest papers by meaning

• CoP: Agentic Red-teaming for Large Language Models using Composition of Principles (NeurIPS 2025 · similarity 0.89)
• AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration (NeurIPS 2025 · similarity 0.88)
• UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning (ICML 2025 · similarity 0.88)
• DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints (AAAI 2025 · similarity 0.88)
• TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration (CVPR 2026 · similarity 0.88)
• RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents (ICLR 2026 · similarity 0.88)