Research map / Large language models / Agents, code and math
BlueCodeAgent: A Blue Teaming Agent Powered by Automated Red Teaming for CodeGen AI
Chengquan Guo, Yuzhou Nie, Chulin Xie, Zinan Lin, Wenbo Guo, Bo Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add constitutions summarized from red-teaming knowledge (ablation) | BlueCodeAgent (GPT-4o) · SecCodePLT vulnerable code detection | F1: 0.64 -> 0.66 | Improved |
| add dynamic testing on top of constitutions (ablation) | BlueCodeAgent (GPT-4o) · SecCodePLT vulnerable code detection | False positives: 120 -> 109 | Improved |
| add dynamic testing on top of constitutions (ablation) | BlueCodeAgent (Claude) · SecCodePLT vulnerable code detection | False positives: 62 -> 50 | Improved |
| add dynamic testing on top of constitutions (ablation) | BlueCodeAgent (GPT-4o) · SecCodePLT vulnerable code detection | F1: 0.66 -> 0.68 | Improved |
| add dynamic testing on top of constitutions (ablation) | BlueCodeAgent (Claude) · SecCodePLT vulnerable code detection | F1: 0.76 -> 0.77 | Improved |
| add dynamic testing without constitutions (ablation) | GPT-4o direct prompting · SecCodePLT vulnerable code detection | F1: 0.64 -> 0.64 | No clear effect |
| add constitutions to dynamic testing (ablation) | GPT-4o with dynamic testing · SecCodePLT vulnerable code detection | True positives: 112 -> 128 | Improved |
| provide similar knowledge code examples instead of constitutions (ablation) | BlueCodeAgent (GPT-4o) · SecCodePLT vulnerable code detection | F1: 0.65 vs 0.66 | Hurt |
| provide similar knowledge code examples instead of constitutions (ablation) | BlueCodeAgent (Claude) · SecCodePLT vulnerable code detection | F1: 0.78 vs 0.76 | Mixed |
| use seen-risk knowledge instead of unseen-risk knowledge (ablation) | BlueCodeAgent · Bias instruction detection | F1 improvement: 0.38 vs 0.31 | Improved |
| use seen-risk knowledge instead of unseen-risk knowledge (ablation) | BlueCodeAgent · Malicious detection (RMCbench) | F1 improvement: 0.34 vs 0.23 | Improved |
| use seen-risk knowledge instead of unseen-risk knowledge (ablation) | BlueCodeAgent · Vulnerable code detection | F1 improvement: 0.09 vs 0.04 | Improved |
Closest papers by meaning
- RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents (ICLR 2026 · similarity 0.92)
- RedCode: Risky Code Execution and Generation Benchmark for Code Agents (NeurIPS 2024 · similarity 0.88)
- MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.87)
- AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration (NeurIPS 2025 · similarity 0.86)
- Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (NeurIPS 2025 · similarity 0.86)
- Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
