Research map / Multi-agent systems and games / Cooperative multi-agent RL
Exploring the Efficacy of Multi-Agent Reinforcement Learning for Autonomous Cyber Defence: A CAGE Challenge 4 Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CAGE Challenge 4, an open-source multi-agent RL cyber defence environment for an enterprise network, and benchmarks top competition agents (mostly heuristics) under generalisation scenarios.
Problem
Single-agent DRL autonomous cyber defence scales poorly to large enterprise networks and creates a single point of failure, yet few environments exist for studying multi-agent RL (MARL) defence against adversaries.
Approach
A configurable multi-agent enterprise network environment (CC4, built on CybORG) where one blue agent per subnet defends against red agents, with a public competition used to benchmark diverse blue agents across modified scenarios; the finding is that heuristic agents outperform MARL submissions.
Key results
- Heuristic agents outperformed all MARL-based submissions across experiments; Team UC heuristic scored -113 ± 35 on CC4 versus -193 ± 84 for the Team Cybermonic MARL agent.
- Under Stealthy Red, Team UC scored -534 ± 347 and Team Cybermonic -996 ± 482 ... the Team Punch heuristic scored -837 ± 203 under Aggressive Red versus -654 ± 244 for Cybermonic MARL.
- All submitted MARL solutions failed to learn to determine network configuration per episode, with Team UC's invalid-action-free observation improving its score under Constant Network Size (-101 ± 36).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a heuristic-based agent rather than a MARL-based agent | CC4 submissions · CC4 and four modified scenarios | average cumulative reward per episode | Helped |
| Fix the number of users and servers to their maximum value | CC4 agents · Constant Network Size | average cumulative reward per episode | Mixed |
| Increase the rate at which green agents trigger phishing attacks | Blue agents · Increased Phishing | average cumulative reward per episode | Hurt |
| Make red agents stealthier | Blue agents · Stealthy Red | average cumulative reward per episode | Hurt |
| Reduce red action durations to one timestep | Blue agents · Aggressive Red | average cumulative reward per episode | Hurt |
| Increase red agents’ ability to detect decoys | Blue agents · Improved Decoy Detection | average cumulative reward per episode | Hurt |
Surprising findings
- Heuristic-based agents outperformed all MARL-based submissions across the top submissions from fifteen participants.
- Team Punch's simple analyse-and-restore heuristic performed much better than its RL agents.
Limitations
- MARL agents failed to infer network configuration at episode start, causing invalid actions.
- Evaluated only a single best agent per top-four team, limiting generality of conclusions.
- Environment is simulated and not validated on real systems.
Experimental setup
Datasets: CAGE Challenge 4 environment
Benchmarks: CC4 Constant Network Size Improved Decoy Detection Increased Phishing Stealthy Red Aggressive Red
Metrics: average cumulative reward per episode
Baselines: Team UC heuristic Team Lancer heuristic Team Punch heuristic Team Cybermonic MARL
Builds on: CybORG PPO CAGE Challenges
Closest papers by meaning
- AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning (ICML 2026 · similarity 0.87)
- Robust Communicative Multi-Agent Reinforcement Learning with Active Defense (AAAI 2024 · similarity 0.86)
- Foundations of Multi-Agent Learning in Dynamic Environments: Where Reinforcement Learning Meets Strategic Decision-Making (AAAI 2025 · similarity 0.86)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.86)
- Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR 2024 · similarity 0.85)
- FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.