Research map / Large language models / Language, safety and interpretability
CoP: Agentic Red-teaming for Large Language Models using Composition of Principles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoP, an agentic red-teaming framework where a Grok-2 red-teaming agent composes human-provided jailbreak principles into single-turn jailbreak prompts, judged by GPT-4 and evaluated on HarmBench.
Problem
Automated jailbreak red-teaming of LLMs is inefficient and has limited effectiveness against highly aligned models (e.g., Llama-2, Claude-3.5 Sonnet), and existing methods lack transparent, extensible strategy composition.
Approach
An LLM red-teaming agent selects and simultaneously applies multiple human-written red-teaming principles (e.g., Expand, Phrase Insertion, Style Change) to form Compositions-of-Principles, iteratively refined with a dual judge scoring jailbreak effectiveness and similarity to the original query.
Key results
- CoP reaches 72.5% ASR on Llama-2-70B-Chat, where all baselines remain below 50%.
- CoP achieves 60.0% ASR on OpenAI O1 and 38.0% on Claude-3.5 Sonnet (on 50 sampled queries), with baselines near zero on Claude.
- CoP needs about 1.5 queries on average for successful jailbreaks on GPT-4-1106-Preview versus 5.63 for AutoDAN-Turbo and 12.11 for PAIR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoP instead of AutoDAN-Turbo | Jailbreak attacks on Gemini · Gemini | ASR: 66.30 -> 78.00 | Helped |
| Use CoP instead of AutoDAN-Turbo | Jailbreak attacks on GPT-4-1106-Preview · GPT-4-1106-Preview | ASR: 88.50 -> 88.75 | Helped |
| Use CoP instead of TAP | Jailbreak attacks on Llama-3-8B-Instruct-RR · Llama-3-8B-Instruct-RR | ASR: 26.0 -> 52.0 | Helped |
| Use CoP instead of TAP | Jailbreak attacks on Llama-2-7B-Chat with Llama-Guard-3 · Llama-2-7B-Chat + Llama-Guard-3 | ASR: 12.0 -> 34.0 | Helped |
| Use CoP instead of X-Teaming with one turn | Single-turn jailbreak attacks on Llama-2-7B-Chat · Llama-2-7B-Chat | ASR: 4.00 -> 64.00 | Helped |
| Use CoP instead of X-Teaming with five turns | Jailbreak attacks on Llama-2-7B-Chat · Llama-2-7B-Chat | ASR: 64.00 -> 64.00 | No clear effect |
| Use CoP instead of PAIR | Successful jailbreaks on Gemini · Gemini | query count: 6.50 -> 1.357 | Helped |
| Use CoP instead of TAP | Successful jailbreaks on GPT-4 · GPT-4-1106-Preview | query count: 26.08 -> 1.512 | Helped |
| Add the similarity judge to CoP's evaluation (ablation) | CoP · HarmBench | ASR: +12% | Helped |
Surprising findings
- Safety-aligned LLMs used as red-teaming agents refused to generate jailbreak principle compositions for 16 of 400 HarmBench queries (Direct Refusal), requiring an initial seed prompt step.
Limitations
- Relies on proprietary APIs (Grok-2, GPT-4) and several headline results use only 50 sampled queries.
- Many baselines' results are taken from prior reports or unavailable (Rainbow Teaming, AutoDAN-Turbo omitted for O1 and Claude).
Experimental setup
Datasets: HarmBench
Benchmarks: HarmBench JailbreakBench X-Teaming multi-turn comparison
Metrics: attack success rate average queries for successful attacks similarity score
Baselines: GCG-T PAIR TAP PAP-Top5 Rainbow Teaming AutoDAN-Turbo X-Teaming
Builds on: PAIR TAP AutoDAN-Turbo RL-Breaker
Closest papers by meaning
- Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents (ICML 2026 · similarity 0.89)
- There are Levels to It: Red Teaming LLMs with Hierarchical Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration (NeurIPS 2025 · similarity 0.88)
- Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.88)
- RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates (ICML 2026 · similarity 0.88)
- Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.