Research map / Adversarial robustness and security / Attacks on text and graphs
Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Rainbow Teaming uses MAP-Elites-style quality-diversity search with LLM mutator, target, and judge components to generate diverse adversarial prompts across safety, QA, and cybersecurity domains.
Problem
Existing automatic red-teaming methods for LLMs are domain-specific, lack diversity, or trade off attack diversity against success rate, and often need human annotation or white-box access.
Approach
Casts adversarial prompt generation as quality-diversity search over a discrete archive of feature descriptors (e.g., risk category x attack style); an LLM mutator produces candidates from archived "stepping stone" prompts, and a pairwise LLM judge decides archive updates instead of a fixed score.
Key results
- Achieves about 92% ASR on Llama 2-chat 7B and Llama 3-Instruct 8B and 98% on Mistral 7B and Vicuna 7B (GPT-4 evaluation) over 2000 iterations.
- SFT of Llama 2-chat 7B on Rainbow Teaming data reduces ASR from 92%/95% to 0.3%/0.7% (GPT-4/Llama Guard) on held-out prompts while keeping GSM8K and MMLU nearly unchanged.
- On JailbreakBench, Rainbow Teaming finds 66/41 jailbroken prompts/behaviours with Llama Guard versus 14/11 for PAIR with the same mutator, with Self-BLEU 0.51 vs 0.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Rainbow Teaming instead of PAIR with the same mutator LLM | Jailbreak prompt generation · JailbreakBench behaviours | successful jailbreaks / behaviours (Llama Guard): 14/11 -> 66/41 | Helped |
| Use Rainbow Teaming instead of PAIR with the same mutator LLM | Jailbreak prompt generation · JailbreakBench prompts | Self-BLEU: 0.74 -> 0.51 | Helped |
| remove parent-child similarity filter (ablation) | Rainbow Teaming · Llama 2-chat 7B archive | ASR (GPT-4): 0.92 \pm 0.01 -> 0.99 \pm 0.01 | Helped |
| remove parent-child similarity filter (ablation) | Rainbow Teaming · Llama 2-chat 7B archive | Self-BLEU: 0.42 \pm{} 0.01 -> 0.79 \pm 0.04 | Hurt |
| fine-tune on Rainbow Teaming synthetic data | Llama 2-chat 7B · New adversarial archives | ASR (GPT-4): 0.92 \pm{} 0.008 -> 0.003 \pm{} 0.003 | Helped |
| fine-tune on Rainbow Teaming synthetic data | Llama 2-chat 7B · New adversarial archives | ASR (Llama Guard): 0.95 \pm {} 0.005 -> 0.007 \pm {} 0.003 | Helped |
| fine-tune on Rainbow Teaming synthetic data | Llama 2-chat 7B · JailbreakBench | PAIR ASR (Llama Guard): 0.14 -> 0.0 | Helped |
| fine-tune on Rainbow Teaming synthetic data | Llama 2-chat 7B · GSM8K (8-shot, maj@1) | score: 0.224 -> 0.219 | No clear effect |
| fine-tune on Rainbow Teaming synthetic data | Llama 2-chat 7B · MMLU (5-shot) | score: 0.412 -> 0.405 | No clear effect |
| reuse archived questions as stepping stones instead of generating each candidate from scratch | Question-answering adversarial prompt generation · Question answering archive | Mean Fitness: 0.79 \pm{} 0.01 -> 0.91 \pm{} 0.01 | Helped |
| add legacy safety system prompt (ablation) | Llama 2-chat 7B · Safety archive | ASR (GPT-4): 0.92 \pm 0.008 -> 0.51 \pm 0.016 | Helped |
| add helpful system prompt (ablation) | Llama 2-chat 7B · Safety archive | ASR (GPT-4): 0.92 \pm 0.008 -> 0.82 \pm 0.029 | Helped |
Surprising findings
- Score-based preference model using Llama Guard probability reward-hacks the classifier, underperforming under GPT-4 evaluation.
- Adversarial prompts transfer on average at about 50% of original-target ASR, with the highest transfer from Vicuna 7B to Mistral 7B despite Vicuna being fine-tuned from Llama 2.
Limitations
- Features and categories are predefined and fixed, and archive size constrains the number of prompts generated.
- Requires extensive computational resources compared with simpler attacks.
- Cybersecurity CyberSecEval judge overestimates ASR relative to human expert evaluation (0.8-0.94 vs 1.00).
Experimental setup
Datasets: Anthropic Harmless (seed dataset in ablation) JailbreakBench GSM8K MMLU Anthropic Harmless Anthropic Helpful
Benchmarks: JailbreakBench GSM8K MMLU CyberSecEval
Metrics: attack success rate self-BLEU BERTScore ROGUE-L compression ratio mean fitness coverage reward model safety and helpfulness scores
Baselines: PAIR TAP No Stepping Stones Same Cell Mutations Score-based (No Judge)
Builds on: MAP-Elites PAIR QDAIF Llama Guard
Closest papers by meaning
- Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.89)
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs (ICML 2025 · similarity 0.87)
- REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective (ICML 2025 · similarity 0.87)
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (NeurIPS 2024 · similarity 0.86)
- SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Models (AAAI 2025 · similarity 0.86)
- Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.