Research map / Large language models / Language, safety and interpretability
Curiosity-driven Red-teaming for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes curiosity-driven red teaming (CRT), which adds entropy and novelty rewards (SelfBLEU and embedding cosine similarity) to RL-based red-team training to produce more diverse yet effective test cases against LLMs, including LLaMA2-7b-chat.
Problem
RL-based automatic red teaming finds effective test cases but converges to a few repeated prompts, giving low coverage of prompts that elicit toxic responses from target LLMs.
Approach
Cast red-team test generation as curiosity-driven exploration: reward novelty of each generated prompt (low negative SelfBLEU and low embedding cosine similarity to past prompts) together with toxicity reward and an entropy bonus, so the red-team policy keeps finding new effective prompts.
Key results
- RL+Curiosity achieves higher SelfBLEU and embedding diversity than RL and RL+TDiv at comparable toxic-response quality in text continuation.
- On instruction following, RL+Curiosity achieves higher quality than baselines against GPT2-alpaca and Dolly-v2-7B across toxicity thresholds.
- CRT finds 196 prompts that elicit toxic responses from LLaMA2-7b-chat-hf, while none of 16 human participants (100 test cases) exceeded toxicity 0.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add entropy bonus and novelty rewards to RL | Red-team GPT2-137M · IMDb text continuation | test-case diversity | Helped |
| Add entropy bonus and novelty rewards to RL | Red-team GPT2-137M · IMDb text continuation | toxic response rate | No clear effect |
| Add entropy bonus and novelty rewards to RL | Red-team GPT2-137M · Alpaca instruction following (GPT2-alpaca target) | toxic response rate | Helped |
| Add entropy bonus and novelty rewards to RL | Red-team GPT2-137M · Alpaca instruction following (GPT2-alpaca target) | test-case diversity | Helped |
| Add entropy bonus and novelty rewards to RL | Red-team GPT2-137M · Databricks instruction following (Dolly-v2-7B target) | toxic response rate | Helped |
| Add entropy bonus and novelty rewards to RL | Red-team GPT2-137M · Databricks instruction following (Dolly-v2-7B target) | test-case diversity | Helped |
| Use curiosity-driven exploration | Red-team GPT2-137M against LLaMA2-7B-chat-hf · LLaMA2-7B-chat-hf red teaming | toxic prompts found: 196 prompts; 0% | Helped |
| Increase the KL penalty weight (ablation) | RL red teaming · Text continuation | quality | Hurt |
| Increase the KL penalty weight (ablation) | RL red teaming · Text continuation | test-case diversity | Helped |
| Increase sampling temperature (ablation) | RL red teaming · Text continuation | test-case diversity | Helped |
| Add an entropy bonus (ablation) | RL red teaming · Text continuation | embedding diversity | Helped |
| Add an entropy bonus (ablation) | RL red teaming · Text continuation | SelfBLEU diversity | No clear effect |
Surprising findings
- No human pilot participant produced a test case eliciting toxicity above 0.5 from LLaMA2-7b-chat-hf, while the automated method found 196 such prompts.
- Raising the sampling temperature of the red-team model increases diversity but still falls short of curiosity-driven exploration.
Limitations
- Novelty reward weights must be tuned per model or task and may dominate the objective.
- Fewer toxic responses are found on RLHF-tuned LLaMA2 than on non-RLHF Dolly-v2-7B.
- Diversity measured by SelfBLEU and embedding distance is a proxy; the paper suggests a benchmark and EIPO as future fixes.
Experimental setup
Datasets: IMDb reviews Alpaca Databricks Dolly15K Toxigen
Benchmarks: IMDb text continuation Alpaca instruction following Dolly instruction following
Metrics: toxic response percentage (quality) SelfBLEU diversity embedding diversity
Baselines: RL RL+TDiv Zero-shot (ZS) Few-shot (FS)
Builds on: PPO curiosity-driven exploration trlx LoRA Perez et al. red teaming with RL
Closest papers by meaning
- Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints (AAAI 2025 · similarity 0.89)
- Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning (ICLR 2025 · similarity 0.88)
- Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.88)
- There are Levels to It: Red Teaming LLMs with Hierarchical Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates (ICML 2026 · similarity 0.87)
- DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.